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A  REVIEW  OF  MODELS  FOR  COST  AND  TRAINING  EFFECTIVENESS  ANALYSIS  (CTEA) 


EXECUTIVE  SUMMARY 


Requirement: 


No  methodology  currently  exists  for  Cost  and  Training  Effectiveness 
Analysis  (CTEA)  that: 

Is  generalizable  to  all  Army  systems  and  nonsystems ^  0'*'^- 
-  Goes  beyond  the  acquisition  phase  of  systems  to  include  the  analysis 
of  fielded  systems. 

This  project  was  to  identify  how  these  needs  would  be  met  through  exten¬ 


sion,  development,  or  refinement  of  current  methods. 


Procedure: 


A  previous  Litton  literature  search  was  refined  and  extended — especially 
to  include  studies  conducted  to  determine  cost  and  effectiveness  of  Training 
Extension  Course  (TEC).  Since  it  was  determined  that  there  existed  no  model 
that  could  be  modified  to  meet  the  research  objectives,  a  systematic  approach 
to  Training  Effectiveness  Analysis  (TEA)  for  multiple  purposes  was  developed, 
based  on  TRADOC  Regulation  350-4.  This  approach  considers  application  of  the 
following  submodels: 

-  CTEA  for  Developing  Systems.  This  submodel  was  developed  by  Litton  in 
previous  research  for  the  Army  Research  Institute  (ARI)  Field  Unit  at 
Fort  Bliss,  Texas  (Matlick  et  al.,  1980a). 

-  Instructional  System  Development  (ISP).  This  submodel  is  covered 
fully  in  TRADOC  Pamphlet  350-30. 

-  Training  Evaluation  for  Nonsystem  Training.  This  submodel  was  deve¬ 
loped  in  the  current  research  effort. 
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-  Initial  Screening  Training  Effectiveness  Analysis  (ISTEA)  for  Fielded 
Systems.  This  submodel  was  developed  in  the  current  effort  based  on 
TRADOC  Regulation  350-4. 

-  Training  Subsystem  Effectiveness  Analysis  (TSEA)  tor  Fielded  Systems. 
This  submodel  was  developed  in  the  current  effort  based  on  TRADOC 
Regulation  350-4. 

-  Training  Developments  Study  (TPS).  This  submodel  was  developed  in  the 
current  effort  based  on  TRADOC  Regulation  350-4. 


F indings : 

No  current  model  or  methodology  lends  itself  to  CTEA  for  developing  and 
fielded  systems,  and  to  systems  and  nonsystems  training.  Therefore,  a  sys¬ 
tematic  approach  to  TEA  was  developed  based  on  a  family  of  submodels.  Methods 
are  available  for  accomplishing  the  processes  embedded  in  some  of  these 
submodels. (e.g. ,  see  Matllck  et  al.,  1980a,  for  methods  recommended  for 
CTEA  for  developing  systems).  For  other  submodels,  the  required  processes 
have  been  identified,  but  the  development  required  of  the  specific  methods  was 
not  within  the  scope  of  the  current  project. 


Utilization  of  findings: 

These  findings,  especially  the  systematic  approach  to  identifying  the 
appropriate  CTEA  model  for  a  given  problem,  will  be  useful  to  training 
developers  and  researchers  in  this  area.  Detailed  methods  for  performing  some 
of  the  new  submodels  developed  in  this  effort  await  their  application,  and 
subsequent  codification  by  analysts  and  researchers. 


IV 


CONTENTS _ 

Page 


Section  1  -  Introduction .  1-1 

Section  2  -  Literature  Review  .  2-1 

2.1  Purpose .  2-1 

2.2  Types  of  Literature  Reviewed .  2-1 

2.3  Army  Regulations  and  Guidance .  2-2 

2.3.1  Introduction .  2-2 

2.3.2  The  Life  Cycle  System  Management  Model .  2-2 

2.3.3  The  Instructional  System  Development  (ISD)  Methodology  2-6 

2.3.3. 1  Phase  I  -  Analyze .  2-7 

2. 3. 3. 2  Phase  II  -  Design .  2-7 

2. 3. 3. 3  Phase  III  -  Develop .  2-7 

2. 3. 3. A  Phase  IV  -  Implement .  2-8 

2. 3. 3. 3  Phase  V  -  Control .  2-8 

2. 3. 3. 6  The  LCSMM  and  the  ISD .  2-9 

2.3.4  Training  Acquisition  Handbook  .  2-7 

2.3.5  TRADOC  Guidance  for  Training  Effectiveness  Assessment  .  2-13 

2. 3. 5.1  TRADOC  Regulation  350-4  2-13 

2. 3. 5. 2  TRASNA' s  TEA  Handbook .  2-13 

2.3.6  The  Training  Device  Acquisition  Process  .  2-20 

2.4  Current  CTEA  Methods  and  Methodologies  .  2-25 

2.4.1  Prescriptive  Methods  .  2-26 

2. 4. 1.1  Training  Effectiveness,  Cost  Effectiveness, 

Prediction  (TECEP)  .  2-26 

2. 4. 1.2  Training  Efficiency  Estimation  Model  .  2-27 

2.4. 1.3  Training  Consonance  Analysis  .  2-30 

2.4. 1.4  Training  Developers  Decision  Aid  .  2-31 

2. 4. 1.5  Systems  Approach  to  Training .  2-31 

2. 4. 1.6  Coordinated  Human  Resources  Technology  ....  2-32 

2. 4. 1.7  Training  Requirements  Analysis  Model  .  2-33 

2. 4. 1.8  Summary  of  Prescriptive  Models  .  2-35 

2.4.2  Predictive  Models  .  2-35 

2. 4. 2.1  TRAINVICE  I  .  2-36 

2. 4. 2. 2  TRAINVICE  II .  2-36 

2. 4. 2. 3  Analogous  Task  Method  . . 2-39 

2. 4. 2. 4  Training  Requirements  Analysis  (TRA)  .  2-40 


v 


CONTENTS  (Continued) 


Page 


2.4.3*  Empirical  Methods  .  2-41 

2.4.4  Litton  Mellonics  CTEA  Methodology .  2-44 

2.4.5  Definition  of  Effectiveness  In  CTEA .  2-48 

2.4.6  Costing  as  a  Part  of  CTEA  Methodology .  2-51 

2. 4. 6.1  Introduction .  2-51 

2. 4. 6. 2  The  Array  LCC  Matrix .  2-52 

2. 4. 6. 3  Cost  Models  from  CTEA .  2-53 

2.5  Reports  of  Cost  and  Effectiveness  Assessments .  2-55 

2.6  Audio-Visual  Training  Extension  Course  Lessons  Literature 

Review .  2-65 

2.6.1  Background .  2-65 

2.6.2  Effectiveness  .....  .  2-66 

2.6.3  Utilization .  2-67 

2.6.4  Cost .  2-70 

2.6.5  Other  Issues .  2-72 

2.6.6  Abstracts .  2-74 

2.7  Summary  of  Review  Findings  .  2-74 

Section  3  -  A  Systematic  Approach  to  CTEA  for  Varied  System  Situations  3-1 

3.1  Introduction .  3-1 

3.2  General  Model  and  Training  System  Analysis .  3-2 

3.3  CTEA  (TRADOC  Reg.  350-4)  Model .  3-6 

3.4  ISD .  3-6 

3.5  The  Training  Evaluation  Model .  3-8 

3.6  ISTEA .  3-15 

3.6.1  ISTEA,  Objective  1 .  3-15 

3.6.2  ISTEA,  Objective  2 .  3-17 

3.7  TSEA .  3-20 

3.8  TDS .  3-20 

3.8.1  General .  3-20 

3.8.2  TDS,  Fielded  Systems .  3-22 

3.8.3  Training  Device  TDS .  3-24 

3.9  Summary  and  Conclusions . 3-28 

APPENDIX  A.  ABSTRACTS .  A-l 

B.  BIBLIOGRAPHY .  B-l 

C.  ABBREVIATIONS  AND  ACRONYMS .  C-l 


vi 


CONTENTS  (Continued) _ 

Page 

LIST  OF  TABLES 

Table  2-1  CTEA  in  the  LCSMM .  2-5 

2-2  Typical  Skill  Performance  Aids  Package  Contents  .  2-11 

2-3  Types  of  TEA  as  Described  in  TRADOC  Regulation  350-4  .  .  .  2-14 

2-4  CTEA  Roles  and  Responsibilities  as  Assigned  by  TRADOC  .  .  2-18 

2-5  Categories  of  Training  Devices  and  Weapon/Operational 

Systems .  2-20 

2-6  Decision  Points  Requiring  TDS ,  Time-Phased .  2-25 

2-7  CTEA  Processes  and  Candidate  Methods .  2-49 

2-8  A  Comparison  of  Cost  Models  Used  for  CTEA .  2-56 

2- 9  Training  Cost  and/or  Effectiveness  Assessments  Reviewed  .  2-57 

3- 1  Categories  of  Assessment  Types  Within  the  Generic  CTEA  .  .  3-3 

3-2  Data  Elements  Collected  for  ISTEA,  Objective  2  .  3-19 

3-3  Decision  Points  Requiring  TDS,  Time-Phased  .  3-25 

LIST  OF  FIGURES 

Figure  2-1  CTEA  in  the  System  Acquisition  Process .  2-4 

2-2  Skill  Performance  Aids  Development  Process  .  2-12 

2-3  TRADOC  TEA  System  Flow  Chart .  2-17 

2-4  CTEA  Flow  Chart .  2-19 

2-5  Training  System  Acquisition  Schedule  .  2-22 

2-6  Sample  of  Matching  Training  Objective  Characteristics 

with  a  Type  of  Learning  Algorithm .  2-28 

2-7  Sample  of  Air  Force  ISD  Media  Selection  Matrix  .  2-34 

2-8  The  TRAINV1CE  Model .  2-37 

2-9  The  Revised  TRAINVICE  Model,  Narva  (1979)  2-38 

2-10  Operational  Test  Cycle  Model  .  2-42 

2-11  General  CTEA  Model .  2-46 

2-12  Overview  of  Methodology  for  SOTAS  CTEA .  2-59 

2-13  SOTAS  CTEA  Effectiveness  Model  .  2-60 

2- 14  SOTAS  Methodology  Candidate  Formulation  .  2-62 

3- 1  General  Model  of  Training  System  Analysis  .  3-5 

3-2  CTEA  (TRADOC  Reg.  350-4)  Model  for  Developing  Systems  .  .  3-7 

3-3  Overview  of  ISD  Procedures .  3-9 

3-4  Model  of  Training  Evaluation .  3-10 

3-5  ISTEA  Objective  1  (Detect  performance  gap  in  fielded 

systems) .  3-16 

3-6  ISTEA  Objective  2  (Provide  supporting  data  for  CTEA 

of  developing  system)  .  3-18 

3-7  TSEA .  3-21 

3-8  General  Model  of  a  Training  Developments  Study  .  3-23 

3-9  Model  of  a  Training  Device  -  Related  TDS .  3-26 


vii 


SECTION  1 


INTRODUCTION 

At  least  fifteen  years  ago* ,  the  Army  recognized  that  the  fielding  of 
Army  systems  without  due  consideration  to  the  impact  of  the  cost  and  effec¬ 
tiveness  of  training  subsystems  was  neither  economical  nor  efficient*  The 
Army's  Life  Cycle  System  Management  Model  (LCSMM)  (DA  Pamphlet  11-25)  defines 
Che  process  by  which  Army  materieL  systems  are  acquired.  Although  it  was 
designed  to  ensure  consideration  of  all  aspects  of  the  system  (training 
subsystem,  personnel  subsystem,  logistics  subsystem),  the  emphasis  has  remained 
on  the  hardware  subsystem.  Nevertheless,  there  continues  to  be  a  need  for 
data  on  the  cost  and  training  effectiveness  of  both  developing  systems  and 
fielded  systems  so  that  decision-makers  can  make  rationaL  decisions  concerning 
competing  developing  systems,  or  on  how  to  ameliorate  excessive  cost  of  or 
ineffective  training  on  fielded  systems. 

The  Army  Research  Institute  (ARI)  Field  Unit  at  Fort  Benning,  Georgia 
sought  to  develop  a  CTEA  methodology  that: 

Could  be  generalized  to  all  training  for  Army  systems  and  nonsystems, 

-  Would  extend  and  refine  previous  research  in  the  CTEA  area,  and 
Go  beyond  the  acquisition  phase  of  systems  to  include  the  analysis 
of  fielded  systems. 

This  research  effort  builds  on  previous  research  for  the  ARI  Field  Unit 
at  Fort  Bliss,  Texas.  The  former  objective  was  to  provide  Army  analysts  with 
a  performance  guide  for  CTEA  at  each  stage  of  the  acquiritlon  process  (LCSMM) 

*The  first  version  of  DA  Pamphlet  11-25,  Life  Cycle  System  Management  Model 
for  Army  Systems  was  published  in  October  196B,  but  the  problem  was  being 
considered  for  several  years  prior  to  this* 
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of  a  developing  system  (Matlick  et  al.  ,  1980a,  1980b).  For  example,  four  Cost 
and  Operational  Effectiveness  Analyses  (COEA)  are  conducted  during  the  LCSMM. 
CTEA  support  COEA  by  providing  assessments  of  alternative  ways  to  train  and 
achieve  the  desired  operational  effectiveness  of  the  system  as  well  as  pro¬ 
viding  relevant  cost  data.  Thus,  at  least  four  CTEA  are  required  to  support 
these  COEA.  Additional  CTEA  are  needed  to  support  training  development 
decisions.  Depending  upon  the  data  situation,  the  analyst  needs  a  different 
strategy  to  perform  each  CTEA. 

To  define  some  terms,  system  training  and  system  device  indicate  that  the 
training  or  device  is  to  be  used  exclusively  for  one  hardware-oriented  system. 

If  the  device  or  training  is  to  be  used  in  conjunction  with  two  or  more 
systems  or  for  general  military  training,  then,  it  is  called  nonsy stem  (ATSC, 
Training  Device  Requirements  Document  Guide,  1979).  Thus,  training  in  military 
courtesy  is  nonsystem  training.  The  Beseler  CUE/SEE  to  be  used  with  Training 
Extension  Course  (TEC)  tapes  is  an  audio/visual  medium  which  constitutes  a 
nonsystem  training  device. 

Although  CTEA  has  been  used  most  commonly  in  a  generic  sense  to  encom¬ 
pass  all  such  analyses  done  for  developing  or  fielded  systems,  and  for  system 
or  no»  system  training,  TRADOC  Regulation  350-4  (1979),  sets  forth  narrower 
definitions.  In  the  TRADOC  Regulation  350-4,  Training  Effectiveness  Analyses 
(TEA)  system  schema,  the  following  terms  are  used  as  indicated: 

Training  Effectiveness  Analysis  (TEA)  -  overall  rubric. 

Cost  and  Training  Effectiveness  Analysis  (CTEA)  -  conducted  for 
developing  systems  In  the  acquisition  phase. 

Initial  Screening  Training  Effectiveness  Analysis  (ISTEA)  -  conducted  for 
fielded  systems  to  determine  if  there  is  a  performance  gap.  The 
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ISTEA  May  have  a  secondary  function  of  providing  support/input  to  a 
CTEA  (on  a  developing  system). 

Training  Subsystem  Effectiveness  Analysis  (TSEA)  -  conducted  for 
fielded  systems  to  determine  if  an  existing  performance  gap  is  due  in 
whole  or  in  part  to  the  training  subsystem. 

Training  Developments  Study  (TDS)  -  conducted  to  develop  a  fix  for  a 
training  subsystem  deficiency  or  develop  a  more  cost-effective  way  to 
train. 

Total  System  Evaluation  (TSE)  -  conducted  to  arrive  at  cost-effective 
ways  to  eliminate  performance  gaps  not  caused  by  the  training  sub¬ 
system.  Since  the  training  subsystem  is  our  primary  concern  here, 
and  since  it  has  been  eliminated  as  a  cause  for  system  deficiency,  we 
will  not  concern  ourselves  further  with  the  TSE.  Suffice  it  to  say 
that  the  TSE  goes  on  to  look  in  detail  at  the  personnel  subsystem,  the 
hardware  subsystem,  and  the  logistics  support  subsystem. 

Thus,  to  place  these  terms  in  perspective,  we  might  say  that:  A  CTEA 
(generic)  may  be  a  CTEA  (TRADOC  Regulation  350-4),  an  ISTEA,  a  TSEA,  or  a  TDS, 
or  a  combination  of  ISTEA,  TSEA,  and  TDS. 

Within  the  research  and  training  development  communities,  the  term 
Weapon  System  Training  Effectiveness  Analysis  (WSTEA)  also  has  been  in  general 
use  with  respect  to  fielded  systems.  In  the  context  of  TRADOC  Regulation 
350-4  ,  the  WSTEA  was  generally  a  combination  of  ISTEA,  TSEA,  and  TDS* 

With  respect  to  systems,  most  CTEA  (generic)  includes  CTEA  (TRADOC 
Regulation  350-4),  ISTEA,  TSEA,  and  TDS.  The  previous  research  developed  the 
model  for  CTEA  (TRADOC  Regulation  350-4).  Our  objective  in  this  research  was 
to  obtain  or  develop  a  generic  CTEA  model  for  systems  and  nonsystems,  and 
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developing  and  fielded  systems.  Once  the  general  model  was  developed  the 
appropriate  submodels  had  to  be  developed.  These  developments  and  the 
resulting  models  are  described  in  the  final  section  of  this  report.  The 
second  section  summarizes  the  literature  on  cost  and  training  effectiveness 
analysis.  Appendix  A  is  a  31-item  bibliography  with  abstracts  of  selected 
related  literature,  Appendix  B  is  a  full  124-item  bibliography  of  CTEA-related 
literature,  and  Appendix  C  is  a  listing  of  abbreviations  and  acronyms  found  in 


the  CTEA  literature. 


SECTION  2 


LITERATURE  REVIEW 

2.1  Purpose 

This  review  of  current  literature  on  Cost  and  Training  Effectiveness 
(CTEA)  methods  was  undertaken  to  determine  the  state  of  the  art  of  such 
methods.  The  review  was  then  to  form  the  basis  for  deriving  or  developing 
a  generic  method  or  set  of  methods  for  CTEA  to  apply  to  a  broader  set  of 
training  programs,  systems  or  devices  -  including  development  efforts  for  both 
developing  anc'  fielded,  systems  and  nonsy  stems-related ,  training  systems. 

2.2  Types  of  Literature  Reviewed 

Three  areas  of  literature  were  identified  as  a  particular  concern. 

These  were  generic  CTEA,  CTEA  TDS  for  major  training  devices,  and  the  original 
focus — CTEA  for  TEC  lessons.  Therefore,  we  examined  four  general  categories 
of  relevant  literature: 

-  Department  of  Defense/Department  of  Army  directives  and  regulations, 

-  CTEA  methods/methodologies  ,  and 

-  Reports  documenting  CTEA. 

The  review  builds  on  the  literature  reviews  of  Matlick  et  al.  (1980a)  and 
Sassone  (1978).  The  Life  Cycle  System  Management  Model,  and  some  of  the  pre¬ 
scriptive,  predictive,  and  cost  models  reviewed  in  the  former  are  reviewed  also 
herein.  These  documents  have  been  re-examined  to  discover  their  treatment 
of  training  media  in  general  and  devices  and  the  Training  Extension  Courses 
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(TEC)  in  particular.  This  reexamination,  however,  constitutes  only  a  small 
portion  of  the  following  literature  analysis  since  the  focus  of  Matlick  et  al. 
(1980a)  was  specifically  on  CTEA  methods  for  developing  systems.  Most  of  the 
Sassone  (1978)  review  focused  on  literature  germane  to  construction  of  a 
cost-effectiveness  model  for  the  training  program  component  (e.g.  ,  TEC).  This 
present  review  has  encompassed  all  literature  available  on  TEC. 

Applicable  literature  identified  in  computerized  searches  of  National 
Technical  Information  Services  (NTIS)  ,  Defense  Technical  Information  Center 
(DTIC)  ,  Educational  Research  Information  Center  (ERIC),  and  other  library 
holdings  have  been  examined.  A  summary  of  the  findings  in  the  four  areas 
follows . 

2.3  Army  Regulations  and  Guidance 

2.3.1  Introduction.  In  order  to  understand  the  context  in  which 
CTEA  takes  place,  an  understanding  of  basic  Army  and  Department  of  Defense 
directives,  regulations,  and  guidance  is  required.  It  is  from  this  literature 
that  the  requirement  for  formal  CTEA  and  other  effectiveness  assessments  are 
identified.  This  literature  also  provides  a  set  of  basic  definitions  of 
terms  without  which  a  study  of  the  plethora  of  CTEA  documents  would  be 
unintelligible . 

2.3.2  The  Life  Cycle  System  Management  Model.  Evaluation  of  the  cost 
and  effectiveness  of  a  training  system  is  required  at  several  points  in  the 
LCSMM  of  a  major  developing  operational  system.  Additionally,  a  training 
device  requires  evaluation  at  additional  points  in  its  own  developmental 
c>cle;  in  some  cases  a  device  is  evaluated  at  the  same  time  as  the  operational 
s>stem. 
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The  process  through  which  Array  materiel  systems  are  acquired  and  main¬ 
tained  is  described  in  DA  Pamphlet  11-25,  Life  Cycle  System  Management  Model 
(LCSMM) ,  expressed  in  a  119-event  flow  chart  (Department  of  the  Array,  1975)* 
LCSMM  outlines  procedures  for  the  development,  acquisition,  and  management  of 
Army  systems  from  concept  investigation  through  ultimate  disposal  of  obsolete 
systems.  It  covers  coordination  of  combat  development,  research  and  develop¬ 
ment,  production  and  logistic  support,  training  and  personnel  requirements, 
and  actions  required  to  develop  and  maintain  the  system. 

Training  considerations  are  integrated  into  the  LCSMM  through  the  Inte¬ 
grated  Personnel  Support  (IPS)  system  (Department  of  the  Army,  1978).  The 
goal  of  IPS  is  to  ensure  that  personnel-related  issues  are  planned,  developed, 
acquired,  tested,  and  deployed  in  conjunction  with  the  materiel  acquisition 
process.  Such  issues  include  the  number  and  characteristics  of  personnel 
required  to  operate,  support,  and  maintain  the  system,  their  training,  inter¬ 
face  with  hardware,  human  resources  development,  and  other  personnel  factors. 

Matlick  et  al.  (1980a)  have  identified  the  locations  for  CTEA  and  CTEA 
updates  in  the  system  acquisition  process  (Figure  2-1).  The  conduct  of 
CTEA-type  analyses  after  a  system  becomes  operational  is  defined  less  clearly. 
The  LCSMM  events  related  to  CTEA  and  the  issues  they  must  resolve  are  shown  in 
Table  2-1*. 

The  process  is  relevant  to  training  devices  from  two  standpoints. 

First,  all  major  devices  and  simulators  arc  subject  tc  the  same  regulations 
and  directives  as  any  other  Army  hardware  system  in  regard  to  procedures  for 
development,  acquisition,  and  management.  Therefore,  they  follow  as  much  of 

*For  a  detailed  description  of  all  LCSMM  events  and  their  implications  for 
CTEA,  see  Matlick  et  al.,  Cost  and  Training  Effectiveness  in  the  Army  Life 
Cycle  Systems  Management  Model,  pp.  II-l  -  11-15. 
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Figure  2-1  CTEA  in  the  Systen  Acquisition  Process 
SOURCE:  Matlick  et  al.  ,  1980,  p.  II-2 
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LCSMM 

PHASE 

CTEA 

PRIMARY  LCSMM  EVENTS 
YIELDING  DATA  TO 

CTEA 

ISSUES  REQUIRING 
RESOLUTION 

CONCEPTUAL 

I 

•  MENS 

•  INITIATION  OF  TRAIN¬ 
ING  PLANNING 

•  INITIATION  OF  LOGISTICS 
SUPPORT  PLANNING 

•  TRAINABILITY  OF 

BASIC  CONCEPT 

•  COST  OF  TRAINING 

•  TRAINING  PROGRAM 
ELEMENTS  TO  BE  IN¬ 
CLUDED  OR  STUDIED 

I  A 

(update) 

•  LOA 

•  ORGANIZATIONAL  AND 
OPERATIONAL  CONCEPTS 

•  TRAINABILITY  OF 
ALTERNATIVE  CONCEPTS 

•  RELATIVE  COST 
EFFECTIVENESS  OF 
TRAINING  PROGRAMS 

OF  ALTERNATIVE 

CONCEPTS 

DEMONSTRATION 

AND 

VALIDATION 

II 

•  OAP 

•  DT/OT  I 

•  TRAINABILITY 

•  NECESSARY  REVISIONS 

OF  TRAINING  PROGRAMS 

•  RELATIVE  COST 
EFFECTIVENESS 

II  A 
(update) 

•  PQQPRI 

•  DCP(IPS) 

•  TRAINABILITY 

•  TDR 

•  RELATIVE  COST 
EFFECTIVENESS  OF 
REVISED  TRAINING 
PROGRAMS 

\ 

FULL-SCALE 

ENGINEERING 

DEVELOPMENT 

III 

•  NET  PLAN 

•  AP 

•  DT/OT  II 

•  PERFORMANCE  VERSUS 
STANDARDS 

•  PERFORMANCE  VERSUS 
HARDWARE  AND  TRAINING 
PROGRAM.  DESIGN, 
PERSONNEL  SELECTION, 
ETC. 

PRODUCTION 

AND 

DEPLOYMENT 

IV 

(update) 

1 

•  TRAINING  PLAN  UPDATE 

•  DRAFT  TRAINING 

PROGRAM 

•  AP 

•  DT/OT  III 

•  COST  EFFECTIVESS  OF 
REVISED  TRAINING 
PROGRAM 

Table  2-1  CTEA  in  the  LCSMM 

SOURCE:  Matlick  et  al.,  1980,  p.  11-41 
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the  LCSMM  (DA  Para.  No.  11-25)  as  each  set  of  unique  conditions  allows. 

Second,  the  LCSSM  for  a  major  system  controls  and  influences  the  subordinate 
development,  acquisition,  testing,  and  operation  of  all  the  devices  related  to 
its  training  systems.  The  scheduling  and  milestones  of  the  major  end-item 
determine  those  required  for  the  training  devices,  sometimes  creating  serious 
compression  of  their  design  and  development. 

2.3.3  The  Instructional  System  Development  (ISP)  Methodology.  The  Army 
and  all  the  other  services  prescribe  a  similar  sequence  of  procedures  for 
the  development  of  training — the  Instructional  System  Development  (ISD)  Model. 
The  procedures  are  described  in  Interservice  Procedures  for  Instructional 
Systems  Development,  TRADQC  Pamphlet  350-30  (1975).  Training  media  (devices, 
simulators,  and  TEC  Lessons)  are  only  a  part,  albeit  an  important  and  expen¬ 
sive  part,  of  a  complete  system  designed  to  train  a  larger  set  of  related 
tasks  required  by  an  operational  systems,  a  MOS ,  or  a  team. 

The  ISD  procedures  constitute  a  systems  approach  to  training  development 
and  as  such  offer  a  repetitive  process  of  analysis,  design,  verification  and 
revision.  Fundamental  to  ISD  are  rigorous  derivations  of  training  require¬ 
ments  for  job  requirements,  selection  of  instructional  strategies  to  maximize 
training  cost  effectiveness,  and  continual  testing  and  revision  of  the  train¬ 
ing  during  development  until  training  objectives  are  met.  Like  the  LCSMM,  the 
complete,  rigorous  application  of  ISD  presents  problems  when  its  use  is 
attempted  in  a  real  world  environment.  The  ISD  process  takes  place  in  five 
phases:  (1)  Analyze,  (2)  Design,  (3)  Develop,  (A)  Implement,  and  (5)  Control. 

Training  media  should  be  dealt  with  in  the  general  ISD  context.  A  brief 
description  of  the  five  ISD  phases  follows. 
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2. 3. 3.1  Phase  I  ~  Analyze.  During  Phase  I,  emphasis  is  on  the  Job  to 


be  performed.'  The  job  performance  is  analyzed  in  as  a  great  detail  as 
possible;  a  complete  task  list  is  compiled;  tasks  that  require  instruction  are 
selected;  a  job  performance  measure  is  specified  for  each  task  selected  for 
instruction;  existing  analogous  training  is  analyzed;  and  the  most  suitable 
instructional  setting  is  proposed  for  each  task  selected  for  instruction. 

Good  research  at  this  stage  is  vital  to  device  specification  and  development 
because  device  design  depends  on  complete  task  description.  Existence  of  job 
performance  measures  makes  possible  assessment  of  training  system  and  device 
effectiveness. 

It  is  in  the  analysis  of  existing  courses  and  prediction  of  settings  that 
use  of  devices  for  certain  tasks  or  groups  of  tasks  may  first  be  recognized. 

2. 3. 3. 2  Phase  II  -  Design.  Phase  II  is  concerned  with  designing  in¬ 
struction  based  on  the  tasks,  sub— tasks,  performance  measures,  setting  selec¬ 
tion  and  other  data  acquired  during  the  analysis  phase.  The  first  step  is  to 
convert  each  task  selected  for  training  into  a  terminal  learning  objective. 
Second,  test  items  are  designed  to  measure  degree  of  mastery  of  the  learning 
objective.  Third,  research  is  conducted  to  determine  the  degree  of  skill  or 
knowledge  that  may  be  expected  before  training.  Finally,  the  tasks  are 
sequenced  into  a  logical,  cch*?rent ,  structured  instructional  program. 

If  this  step  in  ISD  has  been  carried  out,  it  is  of  great  value  in  device 
development  since  it  provides  empirical  criteria  that  can  be  used  to  measure 
trainees  who  are  trained  to  tasks  on  a  simulator  or  device. 

2. 3. 3. 3  Phase  III  -  Develop.  There  are  five  steps  in  the  ISD  develop¬ 
ment  phase.  First,  the  training  developers  classify  the  learning  objectives 
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by  learning  category.  This  enables  them  to  identify  apropriate  learning 
guidelines.  TJext,  they  apply  a  media  selection  process  to  determine  the 
instructional  package  and  presentation.  The  media  selection  process  should 
take  into  consideration  such  factors  as  learning  category  and  guidelines, 
media  characteristics,  training  setting  criteria  and  costs.  Other  steps 
include  analysis  of  existing  analogous  training  packages,  development  of  new 
materials  and  media  as  required,  and  field  testing  and  revision  of  instruc¬ 
tional  material.  The  TEC  prototypes  are  developed  as  part  of  this  package. 

It  is  in  ISD  Phase  III  that  a  need  for  a  device  may  be  identified  or 
verified.  In  the  media  selection  process,  analysts  consider  devices  and 
simulators  and,  if  found  appropriate,  effective  training  media  for  the  tasks 
under  consideration  will  include  them  in  alternative  programs  subject  to  cost 
and  training  effectiveness  analysis. 

2. 3. 3. 4  Phase  IV  -  Implement.  In  the  implementation  phase,  the  training 
delivery  personnel  take  over  the  newly-designed  program  and  put  it  into 

use.  Classes  are  scheduled,  space  made  available,  staff  trained,  materials 
and  media  acquired,  and  students  assigned.  The  appropriate  agency  conducts 
instruction  and  documents  need  for  improvement  for  each  succeeding  training 
media,  and  the  trainers  make  an  assessment  of  strengths  and  weaknesses  along  with 
the  other  components  of  the  instructional  package.  TEC  prototype  packages  are 
validated. 

2. 3. 3. 5  Phase  V  -  Control.  Phase  V  is  the  ongoing,  continued  evaluation 
and  revision  of  the  training  program.  It  is  the  bridge  that  closes  the  loop 
back  to  other  phases  of  the  system.  Throughout  the  life  of  a  training  system, 
data  on  instructional  effectiveness  and  costs,  and  on-the-job  performance  in 


the  field  may  be  collected  and  analyzed.  When  possible,  outside  evaluation 
teams  will  make  such  assessments.  These  analyses  serve  as  a  quality  control 
of  instruction  and  generate  empirical  data  required  to  justify  system  revi¬ 
sions  . 

As  in  Phase  IV,  the  devices  in  use  are  subject  to  the  same  scrutiny  as 
the  program.  The  continuous  data  gathering  on  and  evaluation  of  the  device 
serves  as  a  basis  to  refine  and  improve  it,  cost  it  accurately,  phase  it  out, 
and  provides  a  data  base  for  design  and  development  of  new  devices. 

2. 3. 3. 6  The  LCSMM  and  the  ISP.  The  ISD  process  defines  training  acquisi¬ 
tion  events  and  activities.  However,  these  events  and  activities  occur  and 
are  dealt  with  within  the  context  of  the  overall  system  acquisition 
process  governed  in  the  Army  by  the  LCSMM.  The  LCSMM  Is  applicable  to 
acquisition  initiatives  directed  toward  Improving  fielded,  operational  sys¬ 
tems,  processing  developmental  systems,  and  procuring  new  systems.  A  system 
may  be  a  new  or  existing  training  system.  It  may  be,  and  is  usually  thought 
of  as  being,  a  major  weapon  or  hardware  system.  For  such  systems,  a  complete 
instructional  system  must  be  developed. 

Analysis  by  CORADCOM  (1978)  has  structured  training  events  and  activities 
in  the  context  of  the  LCSMM  as  has  TRADOC  in  TRADOC  Regulation  600-4  (1978). 
The  two  processes,  LCSMM  and  ISD,  are  compatible  although  not  congruent.  ISD 
relies  on  accurate  task  analysis  and  a  great  deal  of  empirical  data  to  achieve 
maximum  utility.  Therefore,  it  is  better  suited  to  design  and  revision  of 
training  for  jobs  that  already  exist.  When  IDS  is  used  within  the  context  of 
the  LCSMM  of  a  new  hardware  system,  real-world  data  such  as  well-defined 
task  lists  and  performance  measures  do  not  exist  to  guide  the  ISD.  However, 
the  general  sequence  of  events  of  both  models  applies  to  training  systems 
acquisition.  This  special  acquisition  process  is  next  discussed. 
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2. 3. A  Training  Acquisition  Handbook.  DARCOM  and  TRADOC  have  provided  a 


consolidated  -set  of  information  and  guidelines  covering  training  requirements 
within  the  acquisition  process  in  a  jointly  prepared  Training  Acquisition 
Handbook  (CORADCOM,  1980).  This  handbook  reviews  the  ISD  process,  the 
LCSMM,  and  training  acquisition  in  the  LCSMM  context.  TEC  lesson*  as  well  as 
training  devices  are  identified  as  part  of  the  media  selection  process.  The 
development  of  TEC,  part  of  the  extension  training  material,  is  keyed  to  the 
technical  manuals.  This  is  a  subordinate  step  in  the  Skill  Performance  Aids 
development  process  (formerly  termed  Integrated  Technical  Documentation  and 
Training  or  ITDT) .  The  handbook  advises  that  a  set  of  material  will  be 
developed  for  each  duty  position.  Table  2-2  shows  the  contents  of  a  typical 
package.  The  handbook  also  states  that  the  lesson  formats  are  modeled  on  the 
TEC  format  that  provides  for  three  instructional  media  formats:  (1)  audio/ 
visual,  (2)  written,  and  (3)  audio.  Use  of  formats  other  than  written  must  be 
justified  by  a  media  analysis.  A  general  model  of  the  Skill  Performance  Aids 
package  development  is  shown  in  Figure  2-2. 

Training  device  acquisition  is  considered  of  major  importance.  The 
handbook  treats  it  in  a  separate  major  section,  Chapter  Six,  Training  Device 
Acquisition.  A  review  of  regulations  and  guidelines  related  to  devices  will 
be  discussed  later  in  this  report. 


*Unless  otherwise  identified,  TEC  lesson  will  be  applied  to  those  audio/visual 
extension  course  lessons  provided  on  8mm  film  and  viewed  on  the  Beseler 
CUE/ SEE. 
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Table  2-2  Typical  Skill  Performance  Aids  Package  Contents 


Documentation  (TMs) 

-  Maintenance  Manuals  (New  Look  format)* 


TM  9-xxx-xxx-20 : 
TM  9-xxx-xxx-30 : 
TM  9-xxx-xxx-40: 

Operator  Manual  (New  Look 

TM  9-xxx-xxx-lO: 


Organizational  Maintenance 
Direct  Support  Maintenance 
General  Support  Maintenance 

format)* 

Operator's  Manual 


Training  (ETM) 

-  Training  Manager's  Guide  (TMG) 

-  Student  Guide  (SG) 

-  Lesson  Administrative  Instructions  (LAI) 

-  Student  Lesson  Sheet  (SLS) 

-  Lesson  Content  Materiel  (Media  Options) 

Audio/Visual 

Written 

Audio 

Computer  Mediated  (CAI,  CMI,  etc.) 
Training  Devices  and  Simulators 


♦Characterized  by  highly  Illustrated,  simple  to  read,  step-by-step  task  per¬ 
formance  instructions  organized  for  use  by  both  entry-level  and  experienced 
personnel.  (Source:  CORADCOM,  1978,  p.  4-3) 
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Figure  2-2  Skills  Perfonuince  Aids  Development  Process 
_  SOURCE:  CORADCOM,  1978,  pp.  4-6 


2.3.5  TRADOC  Guidance  for  Training  Effectiveness  Assessment 


2. 3. 5.1  TRADOC  Regulation  350-4.  TRADOC  provides  guidance  for  several 
types  of  training  effectiveness  assessment  in  two  recent  publications.  The 
first,  TRADOC  Regulation  350-4,  Training:  The  TRADOC  Training  Effectiveness 
Analysis  (TEA)  System  (May  1979)  prescribes  policies,  procedures,  and  respon¬ 
sibilities  governing  the  operation  of  the  TRADOC  TEA  System.  This  regulation 
identifies  and  describes  five  types  of  TEA  as  shown  in  Table  2-3.  The  authors 
state  that  if  valid  evidence  of  a  significant  performance  gap  exists,  a  TSEA 
should  be  conducted  in  lieu  of  an  ISTEA.  They  note  the  similarity  between  a 
CTEA  and  a  TDS,  remarking  that  they  are  procedurally  similar  and  heavily  cost 
(economic)  analysis  oriented.  The  TEA  system  flow  chart,  depicting  the 
relation  of  TEA  to  system  life  cycle,  is  shown  in  Figure  2-3. 

The  regulation  delegates  TEA  authority  and  defines  their  roles  and 
responsibilities  among  several  agencies.  The  pertinent  information  unique  to 
CTEA  is  shown  in  Table  2-4. 

Special  mention  is  made  of  TRASANA' s  role  in  developing  a  TEA  handbook. 
This  is  envisioned  as  an  evolutionary  document,  initially  concentrating  on 
concept  and  methodology  with  the  ultimate  goal  of  providing  a  how  to  guidebook. 

2. 3. 5. 2  TRASANA* s  TEA  Handbook.  The  guidebook  referred  to  in  the 
preceding  paragraph  is  now  available  in  draft  form — TRADOC  Training  Effective¬ 
ness  Analysis  Handbook,  TRASANA  (1980).  This  handbook  is  a  procedural  guide 
for  persons  charged  with  conduct  of  TEA.  The  methodology  for  conduct  of  CTEA 
(from  preliminary  to  update  to  final)  is  shown  in  Figure  2-4.  The  text 
describes  the  method  step-by-step  telling  the  analyst  what  to  do  and  in  some 
cases,  how  to  do  it,  and  suggesting  data  sources. 
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Table  2-3  Types  of  TEA  Described  In  TRADOC  Regulation  350-4 


Description 


Cost  and  Training  Effective-  Conducted  during  the  acquisition 

ness  Analysis  (CTEA)  process  In  order  to: 

Insure  that  Training  Development  (TD) 
processes  (ISD  Phases  I,  II,  and  III)  are 
initiated  early  in  the  life  cycle  of 
hardware  systems  and  are  accomplished  both 
in  parallel  and  coordination  with  Combat 
Development  (CD)  processes  during  the 
acquisition  cycle. 

Optimize  soldier  hardware  subsystem 
interface. 


Insure  that  the  appropriate  level  of 
scientific  methods  are  used  in  the  develop¬ 
ment  of  the  training  subsystem. 

Insure  that  all  feasible  training  sub¬ 
system  alternatives  are  considered. 

Optimize  soldier  training  subsystem  inter¬ 
face. 

Recommend  the  preferred  training  alterna¬ 
tive  for  the  preferred  hardware  system 
based  on  cost  and  training  effectiveness. 

Provide  decision-makers  with  more  precise 
information  at  critical  points  in  the 
acquisition  process  concerning  the  Total 
System  comprised  of  the  training,  hardware, 
and  other  subsystems  (TRADOC  Pamphlet 
71-8). 

Initial  Screening  Training 
Effectiveness  Analysis 
(ISTEA) 


Conducted  after  a  system  has  been  fielded 
in  order  to: 


Determine  if  actual  effectiveness  (E^) 
and  design  effectiveness  (E  )  are 
essentially  equal  or  if  a  significant 
performance  gap  exists. 
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Table  2-3  Types  of  TEA  Described  In  TRADOC  Regulation  350-4 


Training  Subsystem 
Effectiveness 


Training  Developments 
Study  (TDS) 


Determine  if  a  cause  and  effect  relation¬ 
ship  exists  between  demonstrated  soldier 
proficiency  and  attitudes  and  trainer 
proficiency  and  attitudes. 

Examine  aspects  of  the  training  environ¬ 
ment  which  are  most  likely  related  to  the 
E^/E^  relationship. 

Conducted  after  a  system  has  been  fielded 
in  order  to: 

Determine  if  existing  significant  perform¬ 
ance  gap  is  caused,  all  or  in  part,  by  the 
training  subsystem. 

Examine  the  training  subsystem  in  detail. 

Relate  soldier,  trainer,  training  environ¬ 
ment,  training  subsystem  and  hardware 
subsystem  factors/variables  to  obtain  high 
resolution  of  problem  areas. 

Examine,  by  excursion,  related  subsystems 
(personnel  and  logistical  support  sub¬ 
systems)  that  may  be  contributing  agents 
to  a  performance  gap. 

Identify  potential  solutions  to  training 
subsystem  problems. 

Usually  conducted  after  a  system  has  been 
fielded  but  also  is  used  preliminary  to 
the  conduct  of  CTEA  for  developing  system 
training  devices  and  nonsystem  training 
devices  which  are  under  separate  Training 
Device  Letter  Requirement  (TDLR).  The  TDS 
is  designed  to: 

Find  the  most  cost-effective  way  to 
fix  training  subsystems  found  deficient 
during  the  conduct  of  a  TSEA. 

Find  the  most  cost-effective  way  to  change 
training  subsystems  which  are  not  deficient 
but  considered  too  costly  or  in  need  of 
revision. 


Table  2-3  Types  of  TEA  Described  In  TRADOC  Regulation  350-4 


5.  Total  System  Evaluation  TSE,  as  it  applies  herein,  Is  derived  from 

(TSE)  TRADOC  Pamphlet  71-8.  It  includes  the 

training  subsystem,  hardware  subsystem, 
personnel  support  subsystems  and  logical 
support  subsystem.  TSE  are  conducted  on 
fielded  systems  when  It  Is  determined  that 
an  existing  performance  gap  Is  not  caused 
entirely  by  the  training  subsystem.  A  TSE 
is  performed  to  identify  the  problem 
areaCs)  causing  such  gaps.  The  emphasis 
of  a  TSE  Includes,  but  is  not  necessarily 
limited  to,  the  personnel  and  logistical 
support  subsystem. 


Figure  2-3.  TRADOC  TEA  System  Flow  Chart 
Source:  TRADOC  Regulation  350-4,  p.  7. 
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Table  2-4  CTEA  Roles  and  Responsibilities  as  Assigned  by  TRADOC 


Agency 
HQ  TRADOC 


Deputy  Chief  of  Staff  for 
Training  (DCST) 

Deputy  Chief  of  Staff  for 
Combat  Developments  (DCSCD) 


U.S.  Army  Training  Support 
Center  (ATSC) 


TRADOC  Systems  Analysis 
Agency  (TRASANA) 


Provides  policy,  direction,  program 
review,  and  study  approval. 


Directs  the  TRADOC  TEA  System. 


In  coordination  with  the  TRADOC  DCST, 
Insures  OMA  and  RDT&E  funds  for  the  CTEA 
portion  of  the  COEA  are  Included  in  the 
programming  of  funds  (DD  Form  1498)  for 
COEA  and  other  related  combat  development 
studies  (TRADOC  Regulation  11-8). 

Functions  as  HQ  TRADOC  point  of  contact 
fcr  TRASANA  in  CTEA  matters  as  they 
pertain  to  the  overall  COEA  effort. 

In  support  of  the  CTEA  effort,  provides  a 
coordination  link  which  facilitates 
TRASANA' s  entry  into  the  TSM ,  HQ  DARCOM 
and  PM  loop. 

Serves  as  the  TRADOC  DCST  point  of  contact 
(POC)  for  proponents  in  matters  relating 
to  CTEA  study  directives  for  developing 
systems  and  nonsystem  training  devices. 

Responsible  for  the  TRADOC  TEA  Handbook 
which  explains  the  how  to  procedure  and 
methodologies  for  each  type  TEA. 

Conducts  Independent  TEA  efforts  as 
directed  by  HQ  TRADOC. 


Proponent  Service  Schools 


Serve  as  TEA  study  proponent 


UPDATE 


2,3.6  The  Training  Device  Acquisition  Process.  The  Training  Acquisition 


Handbook  (U.S.  Department  of  Army,  CORADOM,  1980)  defines  training  devices  as 
panel  displays*  simulators,  part-task  trainers  and  full  crew  trainers.  They 
are  distinguished  from  training  aids  (descriptive  charts,  graphics,  and 
audio/visual  material)  and  training  equipment  (operational  equipment  dedicated 
to  training)*  Major  training  devices  and  simulators  are  recommended  for  use 
primarily  when  critical  subject  matter  is  too  complex  for  verbal,  symbolic,  or 
simple  pictorial  presentation  or  when  it  requires  extensive  hands-on  practice 
to  develop  the  requisite  level  of  skill.  Categories  of  devices  include 
teaching  machines,  models  and  mockups,  hardware  simulator- trainers ,  and  actual 
objects  (component,  assembly,  unit  or  system).  Training  devices  that  use  the 
actual  equipment  induced  to  act  and  react  as  it  would  in  an  operational 
environment  are  called  simulators  (Montemerlo,  1977). 

PM  TRADE  (1979)  classifies  training  devices  by  their  intended  use.  If 
the  device  is  to  be  used  exclusively  for  one  system,  it  is  called  a  "system" 
device.  If  it  is  designed  for  use  in  two  or  more  systems  or  for  general 
military  training,  it  is  called  a  "nonsystem"  device.  A  further  classifi¬ 
cation  results  when  the  system  itself  is  evaluated  —  is  it  fielded  or 
developmental?  The  four  categories  are  shown  in  Table  2-5. 


Table  2-5.  Categories  of  Training  Devices  and  Weapon/Operational  Systems 

WEAPON/OPERATIONAL  SYSTEM 


1  1 

1  1 

FIELDED 

DEVELOPMENTAL  | 

1  T  1 

NONSYSTEM 

NONSYSTEM  j 

|  R  D  |  NONSYSTEM 

FIELDED 

DEVELOPMENT  j 

|  A  E  | 

|  I  V  j 

|  N  E  | 

SYSTEM 

SYSTEM  | 

|  I  C  |  SYSTEM  | 

|  FIELDED 

DEVELOPMENT  j 

|  N  E  |  | 

1 

1  G  1 _  J 

J 

1 

SOURCE:  PM  TRADE,  1979,  p.  42. 
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Futhermore,  a  complex  simulation  trainer  may  have  several  identical  modules 
for  group  trailing  or  several  different  modules  for  simultaneous  team  train¬ 
ing. 

The  unique  training  device  acquisition  process  begins  after  the  develop¬ 
ment  of  initial  training  system  concepts  (if  the  device  is  part  of  a  develop¬ 
ing  system)  or  after  a  need  is  identified  for  a  new  type  device  or  simulator 
and  validated  by  PM  TRADE  or  TRADOC.  This  process  has  been  modeled  by 
CORADCOM  (1980)  in  relation  to  the  Life  Cycle  System  Mangement  Model  (LCSMM) 
of  the  operational  equipment  for  which  its  training  is  designed;  (Figure  2-5) 
and  PM  TRADE  (1979)  in  greater  detail  and  applicable  to  all  types  of  devices. 

First,  the  device  enters  a  period  of  concept  formulation.  Developers  will 
consider:  (1)  tasks  to  be  trained,  (2)  trainee  characteristics,  (3)  alterna¬ 
tive  hardware  configurations,  (4)  analogous  and  new  simulation  technology,  (5) 
degree  of  fidelity  required,  and  (6)  costs.  Usually  several  alternative 
device  concepts  will  be  developed.  As  work  in  this  concept  formulation  period 
progresses,  the  designated  developers  assisted  by  manufacturers,  trainers, 
users,  behavioral  scientists  and  other  subject  matter  experts  will  produce  one 
or  more  concepts  for  cost-effective  devices.  The  proposed  devices  may  be 
theoretical  designs  or  existing  devices.  They  will  be  assessed  by  a  prelimin¬ 
ary  type  of  cost  and  training  effectiveness  analysis  (CTEA).  U.S.  Army 
Training  Support  Center's  (ATSC)  Guide  to  Training  Effectiveness  Analysis 
for  Training  Devices  (1980)  defines  this  CTEA-type  study  as  a  Training  Deve¬ 
lopment  Study  (TDS)  and  requires  that  a  formal  study  be  made  and  a  report 
written  for  all  devices  with  medium/high  development  risk  during  the  concept 
pha  se.  Based  on  the  TDS,  the  designated  developers,  ATSC  and  PM  TRADE,  will 
decide  whether  to  proceed  with  developme  it  of  a  new  device  or  purchase  of 
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Figure  2.5  Training  Device  Acquisition  Schedule 


commercially  available  equipment.  If  an  existing,  inexpensive,  low  risk 
device  exists t  the  decision  probably  will  be  to  procure  it  by  issuing  a 
Commercially  Available/Fabricated  Training  Device  Requirement  (CAFTDR)  docu¬ 
ment.  This  initiates  procurement  along  the  desired  time  schedule  for  use.  No 
further  development  processes  are  required. 

If  there  is  clear-cut  evidence  to  support  device  specifications  and  cost 
and  developmental  risks  are  relatively  low,  a  training  device  acquisition  may 
proceed  following  preparation  of  a  Training  Device  Requirement  Document  (TDR) 
or  Training  Device  Letter  Requirement  (TDLR).  A  preliminary  TDS  with  cost 
estimates  and  potential  effectiveness  assessment  accompanies  the  TDR/TDLR. 
Following  Department  of  Array  approval,  DARCOM  will  fund  the  device  acquisition. 

If  the  preliminary  TDS  indicates  medium  to  high  developmental  risk,  high 
cost,  unclear  choice  among  alternatives  or  other  substantial  unresolved 
issues,  the  conceptual  phase  closes  with  the  preparation  of  a  Training  Device 
Letter  of  Agreement  (TDLOA).  A  working  document,  the  TDLOA  states  the  issues 
to  be  considered  in  order  to  validate  the  device  concept  before  development 
can  take  place. 

An  update  TDS  is  undertaken  during  the  validation  phase.  It  will  compare 
the  alternative  device  concepts  or  if  there  is  only  one  concept,  it  will  be 
compared  to  a  baseline  of  training  without  the  device.  The  working  group 
conducting  the  TDS  will  evaluate  (1)  how  well  critical  tasks  are  taught,  (2) 
the  potential  for  transfer  of  training  to  the  actual  equipment,  (3)  occurrence 
of  negative  training  or  difficulties  in  use,  and  (4)  any  evidence  of  poor 
design  of  equipment,  training  program  or  device.  With  update  of  costs  includ¬ 
ed,  this  update  TDS  will  be  annexed  to  a  formal  TDR.  If  the  TDR  is  approved, 
the  device  enters  the  Full  Scale  Engineering  Development  (FSED)  phase. 
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During  the  FSED  of  the  device,  the  TDR  is  converted  to  design  specifi¬ 
cations  and  procurement  documents,  a  contractor  is  selected,  and  a  prototype 
device  is  produced.  If  the  device  is  developed  in  conjunction  with  a  major 
system,  its  design  freezes  with  that  of  the  prime  system.  Additionally,  the 
prototype  must  be  ready  for  testing  at  the  time  of  the  prime  system  OT  II  (AR 
1000-1978).  During  this  phase,  the  final  TDS  is  made.  It  Includes  data  from 
the  OT  II  and  addresses  the  same  issues  as  the  previous  TDS.  At  this  time, 
the  analysts  are  able  to  make  the  assessment  based  on  empirical  measures  of 
effectiveness  rather  than  predictive  analytical  measures.  They  are  able  now 
to  make  reliable  cost  effectiveness  determinations.  If  no  adverse  or  un¬ 
resolved  issues  remain,  a  training  device  development  acceptance  In-process 
review  (DEVA  IPR)  is  held.  This  is  the  final  in-process  review;  it  determines 
whether  full-scale  production  should  take  place.  If  the  DEVA  IPR  is  favor¬ 
able,  the  device  advances  to  the  next  phase.  This  nhase  Is  production  and 
deployment.  The  device  is  now  part  of  the  Army  inventory. 

Major  generic  or  nonsystem  devices  and  simulators  (those  at  high  develop¬ 
mental  risk,  costing  over  $3  million  and  initiated  by  a  TDLOA)  follow  the 
general  Army  LCSMM.  CTEA  may  be  necessary  only  if  a  training  system  must  be 
designed  to  manage  the  simulator  or  device  and/or  a  unique  training  system  set 
up  to  train  personnel  to  teach,  operate,  and  maintain  it.  The  required  COEA 
of  the  LCSMM,  since  they  measure  cost  and  effectiveness  (in  this  case  training 
effectiveness),  serve  both  as  COEA  and  CTF.A. 

When  the  training  device  or  simulator  does  not  undergo  the  formal  CTEA 
or  COEA  process,  the  training  developers  usually  conduct  training  develop¬ 
mental  studies  (TDS)  as  input  to  requirements  documents  for  low  risk,  less 
expensive  devices  or  as  input  to  training  system  CTEA  and  operational  system 
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COEA.  A  TDS  is  required  (ATSC,  1980;  TRADOC  Circular  350-4,  1979)  before  each 
device  development  decision  point.  Each  of  these  TDS  requires  an  assessment 
of  potential  device  training  effectiveness  as  well  as  a  determination  of 
non-quantif iable  benefits  and  costs.  The  ATSC  Guide  (TRADOC,  1980)  requires 
that  most  TDS  predict  effectiveness  of  proposed  devices  by  comparing  them  with 
the  present  methods  of  training  similar  tasks.  Table  2-6  displays  the  require¬ 
ments  for  conduct  of  TDS. 

Table  2-6  Decision  Points  Requiring  TDS  Time-Phased 


|  TUS  Type 

Life  Cycle  Phase 

Decision/Document  | 

Requiring  TDS  Input j 

|  Preliminary 

Conceptual 

TDLOA  (full  LCSMM)  | 

Validation 

TDR/TDLR  (partial  | 

LCSMM)  | 

|Update  of  Preliminary 

Validation 

TDR/TDLR  (full  | 

j( follows  DT  I/OT  1) 

LCSMM)  | 

|  Final  (follows  DT  11/ 

Full  Scale 

DEVA  IPR  | 

j  OT  II) 

Development 

2 . 4  Current  CTEA  Methods  and  Methodologies* 

The  major  guidance  for  evaluating  training  effectiveness  is  contained  in 
the  literature  discussed  in  the  preceeding  section  and  such  documents  as 
TRADOC  Circular  70-1,  Training  Device  Development  (1979);  TRADOC  Regulation 
11-8,  Cost  and  Operational  Effectiveness  Analysis  in  Materiel  Acquistlon 


*For  a  complete  review  of  these  methods,  see  Matlick  et  al.,  1980a.  Example 
of  their  application  may  be  found  in  Matlick.  et  al.,  1980b. 
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Process;  and  TRADOC  Pamphlet  71-10,  Cost  and  Training  Effectiveness  Analysis. 
In  addition,  a  number  of  CTEA  methodologies  currently  exist,  the  result  of  a 
great  deal  of  research  in  all  the  services.  They  may  be  categorized  as 
primarily  prescriptive — they  prescribe  suitable  training  programs  analytically 
where  none  exist;  predictive — they  predict  the  effectiveness  of  training 
programs  or  training  program  elements  analytically;  or  empirical — they 
evaluate  training  programs  or  program  elements  using  real-world  performance 
data*  Usually  these  methods  treat  effectiveness  and  cost  as  separate  modules, 
combining  them  in  C/E  ratios  as  a  final  step*  Representative  models  and 
methods  will  be  discussed  in  the  following  paragraphs  of  this  section* 

2*4.1  Prescriptive  Methods*  The  first  set  of  methods  might  be  termed 
prescriptive.  They  prescribe  a  training  program  (or  program  elements  such  as 
media,  context,  devices)  based  on  task  characteristics,  earlier  analogous 
systems  or  other  means. 

2. 4. 1.1  Training  Effectiveness,  Cost  Effectiveness  Prediction  (TECEP). 

One  of  the  most  widely  used  analytical  methods  is  TECEP,  a  manual  method 
developed  by  Braby  and  his  associates  and  reported  in  A  Technique  for  Choosing 
Cost-Effective  Instructional  Delivery  Systems,  TAEG  Report  No.  16  (1975).  The 
Army  and  Navy  ISL)  use  TECEP,  and  it  is  also  the  recommended  method  in  the 
manual  sponsored  by  PM  TRADE  (1979).  TECEP  is  designed  for  use  during  the 
conceptual  phase  for  training  program  development. 

The  TECEP  technique  begins  with  a  list  of  training  objectives,  classifies 
those  objectives  according  to  the  type  of  learning  algorithm  required,  selects 
alternative  media  systems  to  support  those  algorithms,  estimates  the  cost  of 
each  alternative  delivery  system,  and  identifies  a  cost-effective  instruc¬ 
tional  delivery  system.  The  technique  is  simple  though  its  developers  caution 
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that  it  is  intended  for  use  by  experienced  training  system  designers. 

TECEP  is  applied  to  select  instructional  delivery  systems  from  train¬ 
ing  objectives  within  ISD.  Each  training  objective  is  matched  with  one  of 
twelve  learning  algorithms,  accomplished  by  comparing  a  task  to  fhn  various 
algorithms  in  terms  of  action  verbs,  behavioral  attributes,  and  examples  of 
objectives  (Figure  2-6).  Once  tasks  and  objectives  have  been  classified  and 
grouped  according  to  the  learning  algorithms,  a  table  is  used  to  select 
delivery  systems  for  each  group  of  tasks  and  objectives.  Alternative  systems 
then  arc  analyzed  for  cost-effectiveness  comparisons. 

2. 4. 1.2  Training  Efficiency  Estimation  Model.  An  Army  model  based  in 
part  on  TECEP  is  the  Training  Efficiency  Estimation  Model  (TEEM)  of  Jorgensen 
and  Hoffer  (1978),  a  major  portion  of  which  concerns  selection  of  media.  This 
is  a  potentially  highly  useful  model,  since  it  is  also  a  predictive  model  of 
effectiveness,  efficiency,  and  cost-effectiveness. 

The  measure,  the  efficiency  ratio,  represents  a  value  composed  of  the 
efficiency  score  of  an  estimated  program  with  real  world  constraints  divided 
by  the  efficiency  score  of  an  idealized  program  with  no  constraints.  Outputs 
of  this  method  include  a  cost-analyzed  training  program.  Input  requirements 
are  task  list  plus  sufficient  knowledge  of  the  weapon  system  to  permit  in¬ 
ferences  about  the  nature  of  stimuli,  responses,  and  feedback. 

The  selection  of  media  (training  devices  and  materials)  is  accomplished 
by  the  TEEM  computer  program  in  which  training  devices  and  materials  have  been 
described  using  the  same  variables  used  to  describe  the  tasks.  Media  with  the 
highest  number  of  mate  s  with  tasks  become  candidates  for  selection. 

The  media  selection  procedure  results  in  stimulus,  response,  and  feedback 
media  for  each  task.  Since  an  array  of  media  is  not  supportable  in  the  real 
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Figure  2-6  Sample  of  Matching  Training  Objective  Characteristics  With  a  Type  of  Learning 

Algorithm  (from  TAEG  Report  No.  16).  Source:  Matlick  et  al.,  1980b,  p.  III-13. 


world,  the  media  set  for  each  functional  group  of  tasks  Is  reduced  to  a 
supportable  number  derived  from  the  matches  between  task  descriptions  and 
media  descriptions.  The  Ideal,  unconstrained  training  program  establishes  the 
base  case  against  which  reduced  media  sets  are  compared.  By  definition,  the 
efficiency  of  the  unconstrained  media  set  is  1.0.  All  reduced  media  sets  have 
an  efficiency  of  less  than  1.0. 

The  TEEM  computer  program  eliminates  the  medium  with  the  lowest  number 
of  matches  across  all  tasks  from  the  media  set  in  each  matrix  (that  Is,  the 
medium  least  useful  for  stimulus  presentation,  receiving  responses,  and 
providing  feedback).  Iteration  of  the  procedure  measures  the  fit  of  all  tasks 
with  all  functions  and  this  measure  divided  by  the  measure  of  the  ideal, 
unconstrained  case  expresses  the  efficiency  of  the  reduced  media  set.  The 
medium  in  each  matrix  with  the  lowest  number  of  matches  across  all  tasks  is 
removed  from  consideration,  and  the  efficiency  of  the  further  reduced  media 
set  is  calculated.  Iterations  of  the  procedure  continue  until  all  media  have 
been  removed. 

Efficiency  ratios  are  plotted  against  the  number  of  the  iteration.  This 
plot  shows  the  iteration  where  the  efficiency  begins  to  drop  off  steeply  and 
the  analyst  uses  it  to  select  a  media  set  that  provides  the  lowest  acceptable 
efficiency . 

Cost  are  determined  by  describing  media  and  methods  in  terms  of  37  cost 
variables,  and  the  cost  data  are  analyzed  by  the  computer  program  (a  variation 
of  the  TECEP  cost  model).  Outputs  are  program  costs. 

Once  costs  have  been  obtained,  a  decision  metric,  a  cost-ef feciency 
ratio  is  obtained  by  dividing  the  cost  by  the  efficiency  value.  The  analyst 
chooses  alternative  methods,  calculates  their  efficiency  values  and  costs,  and 
obtains  their  cost-ef fecicncy  ratios.  The  analyst  could  recommend  the 
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method  with  the  lowest  cost-efficiency  ratio  (the  lowest  cost  for  at  least 
an  acceptable  efficiency),  or  might  recommend  a  method  with  a  slightly  higher 
cos t-ef f  i  r iency  ratio  to  obtain  a  large  increase  in  efficiency  at  a  higher 
cost. 


2. A. 1.3  Training  Consonance  Analysis  (TCA).  TCA,  a  modification  of 
TEEM,  is  another  method  for  estimation  of  effectiveness.  Developed  by  Hawley 
and  Thomason  (1978),  the  TCA  technique  compares  training  alternatives  on  the 
basis  of  task  descriptions  and  the  methods  and  media  employed  to  train  the 
tasks.  Unlike  TEEM,  TCA  uses  the  descriptions  to  indicate  how  close  the 
media/method  combinations  come  to  the  task  description;  that  is,  TCA  yields 
an  indication  of  the  consonance  of  the  task  requirements  and  the  proposed 
training  programs.  Hawley  and  Thomason  further  modified  TEEM  by  adding  the 
diagnostic  concepts  training  deficiency,  training  excess,  and  training 
redundancy .  The  definitions  of  these  diagnostics  are: 

-  Training  deficiency:  A  variable  in  the  task  description  does  not 
occur  in  the  training  description. 

-  Training  excess:  A  variable  in  the  training  description  does  not 
occur  in  the  task  description. 

-  Training  redundancy:  A  variable  in  the  training  description  is 
redundant.  This  occurs  only  if  two  or  more  training  media  and/or 
methodologies  are  combined  in  the  training  description.  The  variables 
common  to  two  or  more  media/methodologies  are  redundant. 

These  factors  explain  the  differences  in  the  efficiencies  of  alternative 
training  programs.  The  smaller  the  number  of  excesses  and  redundancies,  the 
more  efficient  a  training  program  is  in  media  and  methods. 


Training  consonance  ratios  and  diagnostics  provide  a  basis  for  recommen¬ 
dation  of  a  training  program  and  for  improving  the  program.  TCA  is  applicable 
to  estimating  alternative  devices  effectiveness  or  comparing  a  training  system 
with  a  device  to  a  baseline  without  it.  TCA  is  a  computerized  model  and 
accommodates  a  large  number  of  tasks. 

2.4. 1.4  Training  Developers  Decision  Aid  (TDDA).  Pieper  et  al.'s 
Training  Developers  Decision  Aid,  based  on  TECEP  and  TEEM,  has  both  manual  and 
computerized  forms.  It  has  not  been  applied  widely. 

The  four  functional  elements  of  TDDA  are:  (1)  task  description,  (2) 
training  prescription,  (3)  training  hierarchy  and  sequence,  and  (4)  training 
cost.  Tasks  are  described  using  specific  action  verbs  and  the  piece  of 
equipment  used.  Training  is  prescribed  as  to:  learning  algorithms,  stimulus 
media,  response  acceptance  mechanisms,  method  of  instruction,  and  learning 
setting.  Training  hierarchy  is  the  result  of  assigning  tasks  to  resident 
training,  on-the-job  training  or  no  training,  while  sequence  is  the  specifi¬ 
cation  of  the  order  in  which  tasks  are  to  be  trained.  Relative  costs  of 
feasible  training  alternatives  are  established  through  a  cost-rating  technique. 

2. 4. 1.3  Systems  Approach  to  Training.  One  of  the  most  rigorous  models 
was  developed  by  Sugannan  et  al.  (1975) — the  B-l  Systems  Approach  to  Training 
(SAT)  developed  a  training  program  for  the  air  crew  of  the  B-l  strategic 
bomber.  It  applied  the  techniques  of  systems  analysis  to  instructional  system 
development  to  assure  that  the  entire  training  system  would  be  considered 
within  an  orderly  and  complete  process.  ISD  application  is  rarely  as  rigorous 
as  in  the  B-l  SAT  study. 
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2-1  SAT  Identified  training  device  requirements  as  part  of  method  and 
media  selectioh.  Guidance  for  the  program  prediction  and  identification  of 
training  device  requirements  was  taken  from  the  Navy's  Training  Effectiveness, 
Cost  Effectiveness  Prediction  Technique  (TECE?)  (Braby,  Henry,  and  Morris, 
1975),  modified  to  reflect  the  special  requirements  of  the  B-l  SAT. 

Media/method  selection  is  an  aspect  of  the  structuring  and  scheduling 
of  courses,  tracks,  and  instructional  blocks.  The  tool  employed  to  this  end 
is  the  Training  Resources  Analytic  Model,  a  set  of  computer  programs  that 
examine  proposed  training  system  of  resources,  schedules,  and  costs. 

2. 4. 1.6  Coordinated  Human  Resources  Technology.  In  another  computerized 
model,  the  Advanced  Systems  Division  of  the  Air  Force  Human  Resources  Labora¬ 
tory  (Goclowski  et  al.,  1978a,  b,  c)  combined  five  technologies  in  the  weapon 
system  acquisition  process:  (1)  Maintenance  Manpower  Modeling  (MMM),  (2)  ISD, 
(3)  Job  Guide  Development,  (4)  System  Ownership  Costing,  and  (5)  Human 
Resources  in  design  trade-offs.  All  five  are  applied  individually  during  the 
material  acquisition  process  and  have  data  requirements  in  common.  Therefore, 
one  objective  was  to  integrate  and  apply  the  technologies  to  form  a  Coordi¬ 
nated  Human  Resources  Technology  (CHRT). 

Instructional  Systems  Development  (ISD)  within  CHRT  differs  from  ISD  in 
other  contexts;  it  is  coordinated  with  other  human  resource  technologies 
and  draws  its  data  from  a  base  common  to  these  technologies.  It  results  in 
training  concepts  during  the  concept  phase  of  system  acquisition,  a  training 
plan  during  the  validation  phase,  and  a  fully  developed  training  program 
during  the  full  scale  development  phase. 

Job  Guide  Development  (JGD)  results  in  products  that  may  substitute  for 
or  reduce  the  need  for  training.  JGD  at  present  appears  to  be  concerned  with 


maintenance  tasks,  although  guides  for  operational  tasks  are  consistent  with 
the  JGD  concept. 

Impact  analysis,  the  final  CHRT  activity,  is  the  investigation  of  the 
impacts  on  human  resource  costs  of  a  variety  of  system  alternatives.  CHRT 
assigns  human  resources  and  other  systems  ownership  costs  to  system  design, 
maintenance,  operations,  and  support  alternatives,  &o  that  these  costs  may  be 
considered  fully  during  the  early  and  critical  acquisition  decisions. 

2. A. 1.7  Training  Requirements  Analysis  Model.  A  third  computerized  Air 
Force  method  is  the  Digital  Avionics  Information  System:  Training  Require¬ 
ments  Analysis  (Czychry  et  al.,  1978).  The  Training  Requirements  Analysis 
Model  (TRAMOD)  is  one  of  a  group  of  computerized  analytical  models  that  make 
up  a  life  cycle  cost  model.  It  is  a  computerized  means  for  training  design, 
and  makes  design  decisions  rather  than  simply  revealing  the  impacts  of  de- 

f 

cisions  already  made.  TRAMOD  selects  from  an  input  list  the  tasks  to  be 
trained.  It  generates  a  training  plan  consisting  of:  a  place  of  training 
(school  or  OJT) ,  methods  of  instruction  (simulation,  performance,  lecture, 
etc.),  and  media  (simulator,  mockup,  etc.)  Finally,  it  determines  possible 
schedules.  Task  input  includes  values  for  a  number  of  parameters:  (1) 
criticality,  (2)  learning  difficulty,  (3)  frequency,  (A)  psychomotor  level, 

(5)  cognitive  level,  and  (6)  estimated  time  required  to  accomplish  training. 
In  the  early  stages  of  design  of  new  equipment,  data  values  are  obtained  from 
comparable  operational  equipment. 

TRAMOD  considers  cost  but  not  effectiveness.  Given  adequate  task  data, 
it  would  appear  to  be  a  valuable  CTEA  method  for  comparing  training  alterna¬ 
tives  resulting  from  various  constraints  (limits  on  training  time,  equipment 
shortages,  etc.) 
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The  device-relevant  portion  of  this  model  is  in  the  module  that  assigns 
methods  and  media.  TRAMOD  generates  a  method  and  medium  for  each  task  using 
standard  Air  Force  ISD  media  selection.  A  sample  is  shown  in  Figure  2-7. 
Methods  and  media  are  generated  for  technical  training  school  or  on-the-job 
training  contexts. 


Figure  2-7  Sample  of  Air  Force  ISD  Media  Selection  Matrix 


Taxonomic 

Description 

Learning  Training  Objective 

Method/Media 

Psychomotor  1 
( Imitation) 

Identifications 

TTS:  Discussion/Transparencies 

OJT:  Informal  Lecture/ 

Transparencies 

Cognitive  1 
(Comprehens ive) 

Psychomotor  2 
(Manipulation) 

Learning  Perceptual 
Discrimination 

TTS:  Simulation/Training 

Film 

OJT:  Demonstration/Training 

Film 

Psychomoter  3 
(Precision) 

Psychomotor  4 
(Articulation) 

Understanding  Principles 

TTS:  Simulation/Simulator 

OJT:  Performance/Mockups 

Cognitive  2 
(Application) 

Cognitive  3 
(Analysis) 

Learning  Procedural 

Sequence 

TTS:  Performance/Simulator 

OJT:  Performance/Training 

Film 

Cognitive  4 
(Synthesis) 

f  Cognitive  5 
(Evaluation) 

Making  Decisions 

TTS:  Simulation/Simuiator 

OJT:  Performance/Training 

Film 

Psychomotor  5 
( Naturaliza¬ 
tion) 

Performing  Skilled 

TTS:  Peformance/Simulator 

OJT:  Performance/On-Equip¬ 

ment 

Source:  Parker  and  Downs,  1961,  p.  35. 
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2. 4. 1.8  Summary  of  Prescriptive  Models.  Models  such  as  TECEP,  TEEM, 
etc.  are  typical  of  the  models  that  prescribe  elements  of  a  training  system  to 
train  a  set  of  well-defined  tasks.  ISD  appears  to  be  the  most  frequently 
employed,  perhaps  because  It  Is  so  general.  TECEP  Is  more  specific  to  the 
problem  of  training  prescription  and  also  Is  used  widely.  TECEP  Is  used  as 
originally  designed  and  as  a  part  of  other  models  such  as  TEEM  and  the  many 
spin-offs  from  TEEM.  The  refinements  of  TEEM  (TCA,  TDDA,  ATM)  are  too  new  to 
have  been  applied.  The  remaining  models,  such  as  the  B-l  SAT,  have  been 
applied  to  one  or  more  systems  or  subsystems  in  the  Navy  or  Air  Force.  A 
recent  review  of  ISD  methodologies  by  Vineberg  and  Joyner  (1979)  states  that 
current  procedures  for  selecting  media  appear  adequate  if  instructional 
activities  have  been  specified  in  sufficient  detail.  Conversely,  Fink  and 
Carswell  (1980)  identified  technological  gaps  in  personnel  and  training  and 
they  conclude  that  media  selection  techniques  in  general  need  considerable 
improvement.  Addressing  training  devices  in  particular,  they  find  that 
methods  are  lacking  that  could  help  decide  whether  devices  are  needed  at  all 
and  for  decl.ling  the  strategy  for  using  them.  Fink  and  Carswell  also  state 
that,  "the  relationship  between  stages  of  learning  and  training  devices 
requirements  has  not  been  fully  explored."  A  review  of  media  selection 
methods  as  they  pertain  to  device  is  yet  to  be  undertaken. 


2.4.2  Predictive  Models.  When  a  training  program  or  alternative  programs 
have  been  designed,  the  concepts  must  be  evaluated  to  predict  their  training 
effectiveness  for  use  in  a  CTEA.  Of  the  models/methods  discussed  above,  TEEM 
and  TCA  predict  effectiveness  as  well  as  prescribe  an  effective  system.  Some 
other  methods  are  more  specific,  especially  those  designed  to  predict  device 
effectiveness. 
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2. A. 2.1  TRAINVICE  I*.  Wheaton  and  his  associates  have  produced  the 
most  substantive  original  body  of  work  in  development  of  a  systematic  approach 
to  analysis  and  evaluation  of  alternative  device  concepts.  Their  procedure 
for  this  is  the  TRAINVICE  I  model.  TRAINVICE  I  can  evaluate  either  existing 
or  theoretical  devices  and  allow  comparison  of  either  type  or  a  mixture.  The 
model  compares  the  alternatives  by  relative  training  effectiveness  and  cost 
effectiveness.  The  result  is  a  device  effectiveness  score  for  each  alternative. 
Five  separate  analyses  are  required.  These  are: 

-  Task  commonality  analysis, 

-  Physical  similarity  analysis, 

-  Functional  similarity  analysis, 

-  Learning  deficit  analysis,  and 

-  Training  technique  assessment. 

The  basic  model  is  shown  in  Figure  2-8.  This  model  has  been  used  widely.  PM 
TRADE  (1979)  includes  it  in  their  Training  Device  Requirements  Documents 
Guide . 

Wheaton  et  al.  validated  the  model  in  two  field  studies  (August  1976a, 
b).  As  further  application  and  use  occurred,  modifications  to  TRAINVICE  have 
been  proposed. 

2. 4. 2. 2  TRAINVICE  II.  Narva  (1979)  developed  a  rationale  for  a  revised 
TRAINVICE  based  on  experience  gained  through  utilization  of  the  original.  The 
revised  TRAINVICE  is  shown  in  Figure  2-9.  The  analyst  first  assesses  the 
skills  trained  on  the  device,  whether  all  requisite  coverage  occurs,  how  well 
they  fit  the  training  objectives,  and  why  they  are  Included.  Next,  the 
physical  and  functional  ch iracteristics  are  considered. 

*Originally  TRAINVICE,  known  as  TRAINVICE  I  since  the  development  of  Narva's 
TRAINVICE  II  (1979). 
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These  characteristics  are  used  to  rate  the  adequacy  of  devices  to  meet 
ISD/Braby  et  al.  requirements  and  guidelines.  Each  skill  under  consideration 
is  placed  into  an  appropriate  behavioral  category,  and  the  ISD/Braby  matrices 
discussed  earlier  are  applied.  When  the  entire  model  has  been  applied, 
the  result  is  the  derivation  of  an  index  of  predicted  training  device  effec¬ 
tiveness.  The  formula  for  calculating  the  index  follows  a  procedure  discussed 
by  Gagne  et  al.  (1948).  The  index  of  the  original  TRAINVICE  was  based  also  on 
this  work.  Both  indices  rate  devices  overall  between  0  and  1^,  with  2.  attained 
if  the  device  were  to  follow  all  guidelines  perfectly. 

Josefowitz  and  Kochevar  (1980)  have  extended  TRAINVICE  to  generic  trainers. 
Their  effectiveness  index  is  a  function  of  difficulty  and  criticality  score 
and  requires  at  least  two  devices  since  maximum  attainable  score  rather  than 
perfect  fit  is  the  yardstick. 

2. 4. 2. 3  Analogous  Task  Method.  Two  recent  methods  prescribe  training 
media  and  predict  or  estimate  their  effectiveness  by  comparing  them  to  similar 
existing  systems.  These  are  the  Litton  Mellonics'  Analogous  Task  Method  (ATM) 
and  the  Training  Requirements  Analysis  (TRA)  methodology  of  the  Navy's  Mili¬ 
tary  Manpower  versus  Hardware  Procurement  (HARDMAN)  methodology. 

The  Litton  Mellonics  ATM  (Matlick  et  al.,  1980a,  b)  begins  with  a  propos¬ 
ed  list  of  tasks  for  the  developing  system.  Next,  tasks  of  fielded  systems 
that  are  analogous  to  the  target  tasks  are  identified.  The  empirical  effec¬ 
tiveness  data  of  the  analogous  tasks  (e.g.,  percent  trainees  trained  to 
meet  criterion,  average  length  of  time  required  to  train  to  meet  criterion, 
scores  on  performance  tests)  become  the  estimates  of  the  effectiveness  of  the 
new  program  to  train  the  matching  tasks.  ATM  also  will  prescribe  and  predict 
complete  programs. 
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2. 4. 2. A  Training  Requirements  Analysis  (TRA).  The  TRA  (Vaughan, 

1979)  predicts  effectiveness  In  a  way  similar  to  ATM.  First,  an  estimated 
training  system  is  designed  by  a  skilled  training  developer.  A  procedure 
similar  to  ISD,  but  resulting  in  less  detailed  products,  is  used.  Simulators 
and  devices  are  a  principal  item  considered  in  media  selection.  This  estimat¬ 
ed  system  is  called  the  baseline,  and  its  effectiveness  is  predicted  subjec¬ 
tively. 

Second,  a  reference  training  system  is  selected.  Its  function  is  to 
serve  as  a  data  source  for  the  baseline  through  extrapolation.  The  reference 
training  system  may  be  an  existing  similar  system,  portions  of  an  existing 
similar  system,  or  related  training  system  data  gleaned  from  the  literature. 
Data  to  be  acquired  from  the  reference  systems  are  specific  (e.g.,  executed 
student  performance  by  task).  TRA  assesses  probability  of  success  with  a 
variation  of  a  technique  known  as  fault-tree  analysis.  This  technique  pro¬ 
vides  an  indication  of  the  most  likely  points  of  failure  which  could  occur  in 
any  system.  The  analyst  performs  a  step-by-step  description  of  the  various 
combinations  of  occurrences  within  the  system  design  which  could  result  in 
failure.  A  further  benefit  of  the  technique  is  that  it  depicts  the  probable 
failure  event  sequence  which  can  lead  to  failure  of  a  key  learning  outcome. 

On  completion  of  the  fault-tree  analysis,  formulas  are  used  to  determine 
the  strategic  paths  leading  to  the  desired  events.  Through  a  summation 
process,  the  probability  of  success  of  a  given  design  may  be  predicted.  The 
overall  value  of  this  technique  is  that  it  indicates  the  weakest  point  in  the 
design  as  well  as  the  comparative  success  probabilities  of  alternative  designs. 


2-40 


2.4.3  Empirical  Methods.  Empirical  measures  of  training  effectiveness, 

* 

along  with  methods  and  models  of  effectiveness  analysis  that  require  reliable 
real-world  data,  are  used  to  assess  and  compare  device  and  simulator  effective 
ness  after  they  exist  as  manufactured  hardware.  This  may  be  in  the  form  of 
prototypes,  Initial  production  run  equipment,  or  equipment  that  has  been 
operational  and  in  the  inventory  for  many  years. 

The  agency  with  responsibility  for  testing  at  this  stage  is  the  U.S. 

Army  Operation  Test  end  Evaluation  Agency  (OTEA).  OTEA  may  specify  that  a 
school,  user,  or  contractor  will  conduct  requisite  tests  under  their  oversight 
The  OTEA  operational  test  cycle  model  is  shown  in  Figure  2-10.  Their  basic 
test  designing  process  consists  of  three  elements: 

(1)  Development  of  Test  Conditions 

-  Listing  influencing  factors  and  conditions, 

-  Combining  related  conditions, 

-  Structuring  subtests,  and 

-  Determining  number  of  trials  required; 

(2)  Development  of  Data  Requirement 

Derivation  of  measures, 

-  Selection  of  measures,  and 

-  Statement  of  measures;  and 

(3)  Development  of  Analysis  Logic 

-  Required  assumptions, 

-  Comparisons  planned, 

-  Statistical  and  numerical  tools,  and 

-  Non-numerical  analysis. 
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From  this  design  process  emerges  the  test  concept,  which  should  be  as  compre¬ 
hensive  and  rigorous  as  time,  manpower,  and  funds  permit.  The  process  is 
described  in  Operational  Test  and  Evaluation  Methodology  Guide  (U.S.  Array 
OTEA,  1976). 

Finley  and  Strasel  (1978)  have  defined  effectiveness  training  testing  for 
devices  into  categories  that  are  useful  when  looking  at  training  systems  as  a 
whole.  During  the  developmental  phase  there  are  two  types:  (1)  verification 
testing  and  (2)  system  implementation  testing.  Verification  testing  is 
defined  as  testing  sufficient  to  demonstrate  and  verify  that  the  training 
device  as  actually  developed — hardware— fulfills  the  training  requirement  and 
specification  with  respect  to  training  effectiveness.  They  define  system 
implementation  testing  as  testing  sufficient  to  determine  how  to  best  inte¬ 
grate  the  training  device  into  the  training  program  (POI). 

Verification  testing  takes  place  during  the  device  OT  I  or  II  and,  if 
there  is  a  parent  system,  in  conjunction  with  its  OT  II.  This  testing  deter¬ 
mines  if  the  actual  device  as  hardware  is  as  effective  as  predicted.  By  this 
time,  the  training  developers  should  have  prepared  the  Programs  of  Instruction 
(POI),  identified  representatives  of  the  target  population  to  serve  as 
trainees,  and  developed  standards  and  criteria  for  performance  and  training 
effectiveness  assessment. 

System  implementation  testing,  the  second  type  of  test  yielding  empirical 
data,  takes  place  after  the  device  is  fielded.  It  is  usually  performed  by 
the  user  as  Force  Development  Testing  and  Experimentation  (FDTE)  under  Army 
Regulation  71-3,  and  TRADOC  Regulation  71-9.  The  purpose  of  system  implemen¬ 
tation  testing  is  to:  (1)  determine  the  optimal  use  of  the  device  in  actual 
system  training,  (2)  asses  whether  this  is  being  accomplished,  (3)  measure  the 
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deficiency  (if  any),  (4)  cover  the  reasons  for  less  than  planned  usage,  and 
(5)  suggest  measures  to  achieve  optimal  use. 

During  this  type  test,  the  test  agency  should  investigate  various  ways 
to  integrate  the  device  into  institutional  and  unit  training  considering  the 
primary  system  requirements  as  well  as  other  constraint*.  To  be  fully  effec¬ 
tive,  system  implementation  testing  requires  an  evaluation  of  alternative  uses 
contexts,  and  sequences.  CTLA  methods  applicable  to  fielded  systems,  such  as 
Matlick  et  al.  (1980a)  and  Sassone  (1979)  can  be  used. 

Guidance  for  planning,  conducting,  and  documenting  Training  Effectiveness 
Analysis  (TEA)  studies  is  found  in  the  TRADOC  TEA  Handbook  (Draft,  1980).  The 
section  of  Training  Developments  Studies  (TDS)  is  particularly  applicable  to 
devices. 

The  success  of  all  effectiveness  testing  rests  on  well-designed  experi¬ 
ments,  good  criteria  that  can  be  measured  accurately,  and  models  of  conditions. 

2.4.4  Litton  Mellonics  CTEA  Methodology.  The  proliferation  of  models 
and  methods  for  CTEA  has  created  a  body  of  work  that  is  extensive  and  bewild¬ 
ering.  Analysts  charged  with  conduct  of  CTEA  now  require  systematic  classifi¬ 
cation  and  evaluation  of  the  methods. 

Work  in  this  direction  has  been  completed  recently  for  the  ARI  Field  Unit 
at  Fort  Bliss,  Texas,  by  Matlick  and  Associates  (1980a,  b). 

The  objective  of  this  research  was  to  provide  Army  analysts  with  a 
performance  guide  for  CTEA  at  each  appropriate  stage  of  the  LCSMM.  The 
attainment  of  this  objective  required: 

Determination  of  points  in  the  LCSMM  at  which  CTEA  are  needed. 

-  Assessment  of  the  utility  of  existing  methods  for  CTEA  at  each  point. 


-  Adaptation  of  methods  to  each  LCSMM  point  where  CTEA  are  needed. 

-  Development  of  methods  for  areas  that  existing  ones  do  not  cover  or 
cover  inadequately. 

-  Identification  of  input  information  required  and  available  at  each 
point. 

-  Estimation  of  the  impacts  of  missing  or  degraded  Information  on  the 
CTEA. 

-  Synthesis  of  a  general  model  for  CTF.A  in  the  LCSMM. 

-  Production  of  a  CTEA  Production  Guide  for  Army  analysts. 

Examination  of  the  CTEA  analytical  requirements  and  current  methods, 

methodologies  and  CTEA  reports  showed  that  the  analysts  required  the  follow¬ 
ing:  (1)  a  task  list,  (2)  alternative  training  programs,  (3)  comparable 
effectiveness  data  for  each  alternative,  and  (4)  cost  data  for  each  alternative. 

Matlick  et  al.  devised  a  logical  model  to  guide  the  analyst  in  assessing 
the  data  situation  (Figure  2-11).  The  model  leads  the  analyst  through  ques¬ 
tions  concerning  the  availability  of  task  lists,  training  programs  (including 
alternative  programs),  training  effectiveness  estimates,  cost  analyses,  cost 
effectiveness  compat Isons,  and  issues  to  be  resolved.  Depending  upon  the  data 
situation,  the  analyst  needs  a  different  set  of  CTEA  processes. 

None  of  the  CTEA  methods  unearthed  during  the  research  provided  all 
information  required  for  decisions  at  any  point  in  the  LCSMM.  However, 
detailed  examination  of  the  techniques,  steps,  or  processes  within  the  methods 
(i.e.,  the  elements  from  which  the  methods  are  built)  showed  that  each  method 
contributed  to  the  CTEA  job  to  be  performed.  Some  methods  contributed  more 
elements  within  some  data  situations  than  others.  For  example,  informal, 
expert- judgment  methods  in  current  use  generate  task  lists  if  none  are  avail- 
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able.  Formal  analytic  models  for  prediction  of  training  programs,  however, 
are  superior  to  the  judgmental  models  if  task  lists  already  exist.  General 
methodologies  (e.g.,  TRASANA,  1980)  are  likely  to  describe  what  should  be 
done  without  telling  the  analysts  how  to  do  it.  For  example,  they  are  advised 
to  begin  with  a  task  list  but  not  where  to  obtain  the  information  and  how  to 
compile  the  task  list  if  none  is  available  from  the  tasking  organization  or 
manufacturer  of  the  operational  system. 

Therefore,  the  Litton  research  team  designed  strategies  to  guide  the 
selection  of  the  processes  and  to  conduct  the  CTEA.  The  strategies  correspond 
roughly  to  the  most  common  input  data  situations  at  which  the  analyst  starts 
the  CTEA.  These  were  found  to  be: 

(1)  No  task  list  and  no  training  program. 

(2)  Task  list  but  no  training  program. 

(3)  Training  program  but  no  alternatives  and  no  effectiveness  data. 

(4)  Training  program  with  effectiveness  data  but  no  alternatives. 

(5)  Alternative  training  programs  but  no  effectiveness  data  for  all 
alternatives . 

(6)  Training  program  alternatives  and  effectiveness  data  for  all 
alternatives. 

Extant  CTEA  methodology  contained  three  weaknesses.  First,  use  of 
historical  data  was  hampered  by  inadequate  definition  of  analogous  tasks 
(i.e.,  tasks  in  a  fielded  system  functionally  and  behaviorally  similar  to  tasks 
in  the  proposed  system).  Second,  the  existing  methodology  did  not  thoroughly 
address  the  issue  of  trainability ,  one  of  the  estimations  needed  in  the  LCSMM. 
Third,  the  most  thorough  cost  analysis  model  omitted  costs  of  training  in 
units. 
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Methods  were  devised  to  fill  these  methodological  gaps.  Methods  adequate 

to  perform  the  other  required  processes  already  existed  in  the  Army  community. 

Those  recommended  as  process  guides  are: 

# 

-  Training  Efficiency  Estimation  Model  (TEEM),  Jorgensen  and  Roffer,  1978 

-  Training  Consonance  Analysis  (TCA) ,  Hawley  and  Thomason,  1978. 

-  Training  Effectiveness,  Cost  Effectiveness  Prediction  (TECEP),  also 
known  as  TAEG  No.  16,  Braby  et  al.,  1975. 

-  Army  CTEA  Methods  in  current  use:  (1)  DIVAD  Gun  CTEA,  (2)  Improved 
Hawk  (HAWK  PIP)  Training  Development,  (3)  Roland  Training  Development, 
(4)  Improved  TOW  Vehicle  (ITV)  CTEA,  and  (5)  Methods  for  the  Analysis 
of  Training  Devices  and  Simulators  (TRAINVICE  I) . 

Most  of  these  are  described  in  other  sections  of  this  review.  Table  2-7  shows 
the  processes  by  data  input  requirement  type. 

2.4.5  Definitions  of  Effectiveness  in  CTEA.  It  is  not  always  clear  what 
is  implied  by  the  term  'training  effectiveness'.  Most  studies  documenting 
effectiveness  research  have  defined  the  term  but  since  each  one  treats  the 
subject  from  a  unique  perspective,  definitions  are  varied.  Hawley  and  Thoma¬ 
son  (1977)  defined  it  as  follows: 

Training  effectiveness  is  defined  as  the  degree  to  which 
soldiers  or  units  trained  using  a  specified  training  system 
can  attain  or  exceed  mission-related  performance  objectives 
on  performance-oriented  criterion  tests  requiring  application 
of  the  skills  addressed  by  the  training  system. 

Jorgensen  and  Hoffer  relate  it  to  change  in  performance: 

Effectiveness  is  defined  as  the  change  in  performance  from 
a  pre-instruction  level  to  a  post-instruction  criterion. 

The  failure  to  use  job  performance  to  validate  training  was  noted 
by  Orlansky  and  String  (1979).  They  state: 
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...the  effectiveness  of  alternative  methods  of 
instruction  must  be  evaluated  by  comparing  how  well 
personnel*,  ....perform  in  operational  units. 

In  the  case  of  devices  (and  perhaps  by  extension  to  media)  ,  the  basic 

definition  is  that  of  Gagne'  (1962)  who  said: 

When  one  inquires  about  the  effectiveness  of  a  device 
for  training,  one  is  really  asking  about  transfer  of 
learning  to  some  criterion. 

Thus,  rationale  for  use  of  training  devices  rests  on  the  body  of  transfer  of 
training  research. 

The  lowest  criterion  for  the  effectiveness  of  training  is  whether  it  does 
indeed  train  personnel  on  tasks  it  was  designed  to  train.  This  may  be  deter¬ 
mined  by  empirical  measurement  when  operational  and  training  systems,  devices, 
or  prototypes  exist;  or  It  may  be  predicted  at  early  stages.  If  tasks  it  was 
designed  to  train  are  trained ,  then  effectiveness  in  relation  to  alternative 
designs  and  the  operational  equipment  (hands-on  or  on-the-job  training)  may  be 
evaluated  along  other  dimensions. 

Measures  of  effectiveness  of  devices  derive  from  such  a  definition  as 

that  of  Wheaton  et  al.  (p.  80,  1976),  paraphased  as  follows: 

An  effective  training  device  is  one  which  teaches  the 
appropriate  behaviors  in  an  efficient  manner.  To 
satisfy  military  requirements,  it  may  be  well  to  modify 
this  definition  to  include  both  the  achievement  of  task 
standards  and  the  constraint  of  reasonable  cost.  Thus, 
a  cost-effective  device  is  one  which  teaches  the  appro¬ 
priate  behavior  to  standards,  efficiently  and  at  reason¬ 
able  cost. 

It  should  be  noted  that,  t n  order  to  minimize  error,  the  familiar  device 
concepts  of  transfer  of  training  and  of  similarity  or  fidelity  are  not  in¬ 
cluded  in  this  definition.  First,  it  implies  that  a  training  device  is  a 
teaching  machine  rather  than  a  substitute  work  environment  and  that, 
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therefore,  the  learning  rather  than  the  end  performance  is  the  primary  concern. 
Second,  If  the  appropriate  learning  does  take  place,  there  will  be  transfer  of 
training;  that  Is,  the  appropriate  performance  will  also  take  place.  Finally, 
similarity  (physical,  functional,  and  psychological  fidelity)  only  usually, 
net  always,  provides  for  transfer  of  training  and  thus  for  appropriate  per¬ 
formance. 

Hall,  Rankin,  and  Aagard  (1976),  examining  specific  problems  affecting 
Navy  training  evaluation  programs,  distinguished  between  training  effective¬ 
ness  and  job  performance  as  follows: 

...trainine  effectiveness  is  used  exclusively  to  refer  to 
(measures  of)  the  degree  to  which  a  training  course  or 
system  achieves  the  goal  established  for  it.  ...job  per¬ 
formance  is  both  a  measure  of  effectiveness  and  validity. 

Achievement  of  course  objectives  is  a  measure  of  effective¬ 
ness  and  may  be  a  measure  of  validity  to  the  extent  that 
it  Is  correlated  with  performance. 

2.4.6  Costing  as  a  Part  of  CTEA  Methodology* 

2. 4. 6.1  Introduction.  Costing  of  alternative  and/or  recommended  train¬ 
ing  systems  for  weapon  systems  is  an  integral  part  of  the  CTEA  process. 

Maximum  effectiveness  and  benefit  must  often  be  sacrificed  by  decision-makers 
because  of  excessive  cost.  This  is  the  basis  for  inclusion  of  cost  in  every 
CTEA.  In  most  CTEA,  alternative  systems  are  developed  on  the  basis  of  pre¬ 
dicted  effectiveness,  a  comparison  with  an  identified  baseline  made,  and 
costs  are  the  last  consideration  before  a  final  selection  is  made.  This  is 
probably  the  result  of  the  greater  involvement  of  training  specialists  in 
development  of  CTEA  methodology  than  of  cost  analyses. 

*For  abstracts  of  the  methods  of  costing  in  CTEA  methods  and  methodologies 
discussed  in  this  report,  see  Matlick  et  al.,  1980a,  Appendix  B. 
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Another  problem  exists  in  the  costing  approach.  As  pointed  out  In 
Department  of *  Army  Pamphlet  11-5,  costing  may  be  done  using  either  a  top-down 
or  a  bottom-up  approach  (Figure  2-9).  Both  are  valid  approaches,  serve 
different  ends,  and  derive  component  costs  differently.  Department  of  Army 
Pamphlet  11-5  sta.es  that  there  is  a  need  to  establish  a  structure  that 
preserves  and  interfaces  the  two  approaches. 

All  CTEA  cost  methodology  must  be  set  within  and  related  to  the  require¬ 
ments  of  the  LCSMM.  In  order  to  understand  the  CTEA  costing  and  the  framework 
in  which  it  sets,  an  examination  of  the  Army  Life  Cycle  Cost  (LCC)  matrix  is 
required. 


2. 4. 6. 2  The  Army  Life  Cycle  Cost  Matrix  (LCCM).  The  LCCM  gives  the 
approved  cost  element  structure  by  appropriation  and  LCC  category.  This 
particular  structure  is  designed  to  capture  and  display  all  costs  in  the  life 
of  a  major  weapon  system  and  its  components.  Training  cost  is  included  and 
defined  specifically  in  this  model.  It  is  comprised  of: 

(1)  Cost  of  designing  equipment  and  training  and  cost  of  purchasing 
training  material  in  the  Research,  Development,  Training,  and 
Evaluation  (RDT&E)  phase, 

(2)  Initial  training  of  operators,  quantity  purchase  of  equipment  and 
spare  parts,  and  investment  in  facilities  in  the  investment  or 
non-recurring  cost  phases,  and 

(3)  Training  to  replace  personnel  lost  to  the  Army  and  training  ammuni¬ 
tion  and  missiles  in  the  operational  phase. 

A  third  problem  is  one  of  focus — the  analyst  specializing  in  one  area 
is  not  always  able  to  isolate  and  aggregate  all  relevant  costs.  Training  is 
an  outstanding  example  of  such  an  area.  A  CTEA  analyst  who  wishes  to  make  a 
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complete  life  cycle  costing  of  training  must  have  6uch  data  as  ammunition 
and  POL  that  -the  weapon  system  cost  analyst  may  be  costing  In  other  ways  and 
in  other  categories. 

Interface,  therefore,  becomes  a  problem.  This  was  recognized  by  Hawley 
and  Thomason  (1978)  who  recommended  more  definitive  work  be  done  on  the  cost 
models  used  for  CTEA  to  make  their  input/output  compatible  with  the  Army's 
accounting  system. 

The  TRADOC  TEA  Handbook  (TRASANA,  1980)  also  sets  costs  within  the 
framework  of  the  LCSMM.  Costs  are  to  be  estimated  in  the  early  stages  and 
predicted  as  accurately  as  possible  for  input  to  the  final  COEA.  LCC  Esti¬ 
mates  (LCCE)  are  mandated  specifically  for  a  list  of  instructional  and 
unit  categories.  Unfortunately,  a  large  number  of  CTEA  methodologies  were 
developed,  and  many  CTEA  (of  sorts)  were  conducted  before  this  handbook  was 
published  (it  is  still  in  draft). 

2. 4. 6. 3  Cost  Models  From  CTEA.  Since  time  constrains  the  analysts 
charged  with  conducting  CTEA,  a  number  of  methodologies  have  been  designed. 
Unique  cost  models  are  seldom  a  feature  of  these  efforts.  Usually,  existing 
training  cost  models  (LCC  or  other)  have  been  found  that  include  all  the 
elements  believed  required  for  costing  training  systems.  These  have  been 
added  and/or  adapted  to  a  variety  of  methods  designed  to  select  training  for 
the  tasks  and  assessing  the  effectiveness  of  that  training. 

Obviously,  a  unique  and  complete  costing  effort  for  each  CTEA  would  be 
the  best  approach.  Since  time  and  money  are  not  usually  available  for  this, 
good  standard  methodologies  and  models  must  be  employed. 

The  more  complete  these  are,  the  less  danger  there  is  of  the  CTEA  team 
omitting  an  important  element.  This  also  is  the  rationale  for  cost  analysts 
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to  work  closely  with  training  designers  and  evaluators.  It  orevents  gaps 
In  data  required  for  the  system  COCA. 

CTEA  methodologies  and  other  related  studies  examined  show  two  approaches. 
They  may  be  termed  either  comparative  or  comprehensive.  In  the  comparative 
cost  approach,  only  those  areas  where  alternatives  may  be  expected  to  differ 
are  costed.  For  example,  in  designing  training  for  pilots,  training  hours  in 
a  simulator  and  in  the  aircraft  will  be  compared  on  the  basis  of  operational 
costs- -fuel,  maintenance,  etc.  Elements  such  as  number  of  classroom  hours, 
preceding  both  simulator  and  aircraft  training  will  be  found  the  same,  and 
excluded  from  consideration.  This  approach  is  the  method  of  choice  for  CTEA 
conducted  early  in  the  acquisition  process.  It  can  identify  cost  drivers  in  a 
training  system  and  eliminate  enough  alternatives  to  narrow  the  field  to  the 
most  promising  few  on  the  basis  of  excessive  cost.  This  can  work  also  in 
reverse.  If  all  training  alternatives  are  estimated  to  be  of  approximately 
equal  cost,  that  consideration  can  be  secondary  and  time  and  manpower  concen¬ 
trated  on  other  assessment  areas. 

The  second  approach  is  one  of  total  or  LCC.  This  is  a  highly  skilled 
eifort  requiring  experienced  analysts.  In  LCC,  it  is  necessary  to  attempt  to 
capture  all  the  costs  that  can  be  identified,  subjecting  them  to  such  factors 
as  discounting,  inflation,  etc. 

Because  of  the  uncertainties  of  costing  future  years  in  an  inflationary 
economy,  one  research  group,  Pieper  et  al.  (1978),  devised  a  method  of  cost 
classes  and  rankings  to  employ  cost  in  a  cost-effectiveness  assessment  wlthouc 
time-consuming  gathering  and  conversion  of  dollar  estimates. 

Alternative  training  systems  usually  are  costed  on  the  basis  of  facilities, 
personnel,  equipment  (including  media  and  devices)  and  supplies  (including 
^munition).  The  institutional  setting  usually  is  chosen;  occasionally  the 
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unit.  On-the-job  training  costing  is  rare,  although  a  number  of  good  models 
for  deriving  OJT  costs  exist  and  could  be  added  to  the  cost  models  in  use  as  a 
variant  of  unit  training.  (The  distinction  between  OJT  and  unit  individual 
training  in  this  context  is  that  OJT  trainees  have  had  no  prior  institutional 
training  in  their  MOS.  The  others  have  had  some  tasks  trained  in  the  schools 
and  others  in  their  first  unit  assignment.) 

Table  2-8,  displays  a  comparison  of  CTEA  costing  methods  found  in  the 
documents  reviewed. 

2.5  Reports  of  Cost  and  Effectiveness  Assessments 

A  third  category  of  documents  is  relevant  to  the  subject  of  training 
cost  and  effectiveness  evaluation.  These  are  the  reports  of  various  assess¬ 
ments  conducted  on  training  of  fielded  and  developing  systems  and  training 
devices.  This  very  extensive  category  has  been  sampled  for  this  review  (Table 
2-9).  Such  a  review  provides  insight  necessary  to  understand  the  TEA  and  CTEA 
process  as  it  is  currently  used  as  well  as  its  strengths  and  weaknesses. 

During  the  last  decade  emphasis  on  CTEA  has  increased,  the  Department  of 
Defense  and  Department  of  Army  directions  and  guidance  have  evolved,  and,  as 
may  be  expected,  the  resulting  mode  of  assessment  has  changed.  Each  study 
works  from  a  unique  perspective.  The  analysts  have  undertaken  Cost  Effective¬ 
ness  (C/E)  Analysis,  Cost  and  Operational  Effectiveness  (COEA) ,  Cost  and 
Training  Analysir  (CTEA),  Weapon  System  Training/Training  System  Effectiveness 
Analysis/Assessment  (WSTEA,  TSEA)  and  documented  their  work  under  these  and 
many  other  variant  nomenclatures.  Work  has  taken  place  at  all  the  different 
stages  of  the  LCSMM.  Sometimes,  the  effort  has  been  fully  supported  and 
iterated  as  required;  other  work  was  accomplished  with  little  allotted  re¬ 
sources  and  tiiua.  Therefore,  this  body  of  literature  does  not  lend  itself  to 


Table  2-9..  Training  Cost  and/or  Effectiveness  Assessments  Reviewed 


AN/TSQ-73 

Attack  Helicopter-1  Flight  and  Weapons  Simulator 
BT  33 
DIVAD  Cun 

Diagnostic  Rifle  Marksmanship  Simulator 
F-16  Avionics  Intermediate  Shop  Training  Program 
Firefinder 
Flight  Simulators 

Ground/Vehicular  Laser  Locator  Disignator 
Improved  Tow  Vehicle 
LAW 

Light  Weight  Company  Mortar  System 
M16A1  Basic  Rifle  Marksmanship  Training  Program 
M60  Firetrainers 
AF  Photography  Course 
REDEYE 

•*  » 

Remoted  Target  System  Non-System  Training  Device 
Simulator  for  Air-to-Air  Combat 
Stand-off  Target  Acquisitiop  ’Surveillance  System 
STINGER 
TEC 

XM1  System  Training  Device 


generalization.  It  is,  however,  of  great  value  to  the  analyst  who  may  be 
searching  for  examples,  models  or  baseline  data. 

WSTEA  art  probably  the  best  source  of  good  quality,  empirically  derived 
baseline  data.  The  REDEYE  WSTEA  (TRASANA,  1977)  and  TSEA  (TRASANA,  1978) 
illustrate  this.  These  two  studies  contain  data  on  training,  job  perfor¬ 
mance,  attitudes,  cost,  retention  of  training,  soldiers'  characteristics, 
distribution  of  weapons,  operators  and  malntainers,  several  simulators,  and 
other  variables  as  they  pertain  to  the  REDEYE  weapon  system  in  the  Active 
Army,  Army  Reserve  Components,  and  Marine  Corps.  Among  the  objectives  of 
these  studies  was  the  provision  of  baseline  data  for  the  follow-on  air  defense 
system — the  STINGER. 

Highly  relevant  was  the  finding  that  70%  of  soldiers  could  not  memorize 
a  training  aid  called  the  Range  Ring  Profile  (RRP).  A  more  complex  RRP  had 
been  planned  for  the  STINGER.  Looking  at  another  useful  category  of  litera¬ 
ture,  DT/OT  Test  Design  Plans  (TDP) ,  we  find  only  one  issue  addressing  train¬ 
ing  in  the  STINGER  TDP  (OTEA,  1976).  This  issue  is: 

Is  the  proposed  training  package  adequate  to  allow  the  STINGER 

section  to  successfully  accomplish  assigned  tasks? 

Because  of  REDEYE  WSTEA/TSEA  findings,  STINGER  training  plans  were  modified 
and  the  RRP  simplified  to  allow  a  positive  response  on  this  issue. 

The  early  CTEA  were  mostly  likely  to  be  short  and  address  certain  speci¬ 
fic  questions.  They  did  no  follow-up  CTEA  models  or  methodologies.  Such  recent 
CTEA  as  ROLAND  (TRANSANA,  1979),  Ground/Vehicular  Laser  Locator  Designator 
(G/VLLD)  (BDM,  1976-1979),  and  the  Stand-off  Target  Acquisition  Surveillance 
System  (SOTAS)  (ARINC,  1977)  are  examples  of  complete  CTEA  and  serve  as 
excellent  models.  SOTAS,  a  CTEA  conducted  as  input  for  the  final  system  COEA, 
is  typical.  Figure  2-12  displays  the  methodology. 

The  SOTAS  effectiveness  model  is  shown  in  Figure  2-13.  The  study  team 
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used  an  adaptation  of  the  Braby  approach  (TAEG  16  and  TAEG  23)  for  formulating 
candidate  training  systems  (Figure  2-14).  They  note  that  a  derivation  of 
this  method  has  been  used  by  the  Army  Research  Institute  in  the  CTEA  for  the 
AN/TSQ-73  Missile  Minder  (prototype  TEEM).  The  technique  involves  separate 
analysis  of  the  tasks  to  be  learned  and  the  applicable  and  available  training 
delivery  media  (TAEG  23  Guidelines).  Results  of  these  two  analyses  are  then 
combined  and  grouped  to  produce  separate  training  system  candidates.  The 
relative  effectiveness  of  the  candidate  systems  is  assessed  using  a  modi¬ 
fication  of  the  cost  model  that  was  developed  by  Braby  (1975)  and  a  20-year 
life  cycle  was  costed.  Using  these  combined  techniques,  the  SOTAS  study  group 
developed  training  system  candidates,  predicted  their  effectiveness,  and 
ranked  them  through  an  evaluation  of  the  results  of  life  cycle  cost  and 
relative  training  effectiveness  analyses  and  qualitative  considerations  (e.g.  , 
Anny/school  doctrine,  training  device  maintenance  support,  in-field  refresher 
training  support). 

Training  device  and  simulator  COEA  and  CTEA  are  more  numerous  and  precise 
than  operational  systems  reports  to  date.  This  is  in  part  due  to  the  well- 
developed  regulations,  directives  and  guidance  pertaining  to  the  device/simu¬ 
lator  acquisition  process.  Historically,  heaviest  device  use  has  been  in  the 
various  flig.it  simulators.  A  comprehensive  review  of  flight  simulator  C/E 
research  through  the  Mid-70's  was  conducted  by  Orlansky  and  String  (1978). 

They  compared  effectiveness  among  studies  using  a  transfer  effectiveness  ratio 
(TER)  developed  by  Provenmi re  and  Roscoe  (1973).  The  TER  has  wide  potential 
for  assessment  of  any  device  or  training  media  that  substitutes  (at  least  in 
part)  for  operational  equipment.  TER  compares  the  amount  of  flight  time 
saved  to  amount  of  time  spent  in  the  simulator  to  reach  comparable  levels  of 


performance  efficiency 
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Figure  2-14.  SOTAS  Methodology  Candidate  Formulation 
Source:  AH INC  (1977),  p.  A-6. 
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The  Issue  of  simulator  or  training  element  utilization  is  receiving  more 
attention  in  TEA  and  CTEA.  A  training  device  or  medium  may  provide  satisfactory 
transfer  of  training  but  if  it  is  not  used,  or  is  little  used,  it  will  not  be 
cost  effective.  This  is  especially  true  if  device/  medium  acquisition  and/or 
operating  costs  are  also  high.  A  method  for  assessing  CE  with  various  levels  of 
utilization  was  employed  in  the  All—  1  Flight  Simulator  (AH-1  FS)  CTEA  (U.S.  Army 
Aviation  Center,  1979)  (USAAC).  USAAC  found  that  saving  wilj.  result  if  current 
plans  to  integrate  the  AH-1  FS  into  POI  take  place.  These  savings  are  highly 
dependent  upon  amount  of  flight  simulator  time  per  student.  A  further  complica¬ 
tion  is  the  known  requirement  for  simulator  maintenance  downtime.  Therefore, 
the  AH-1FS  study  group  developed  a  series  of  curves  (one  for  each  of  six  train¬ 
ing  sites)  that  related  operating  and  support  costs  per  hour  to  training  hours 
per  year  based  upon  a  range  of  percents  of  time  that  the  simulator  might  be 
available. 

Missing  data  are  compensated  for  in  several  studies.  BDM  (1980)  in  assess¬ 
ing  the  C/VLLD  Trainer  found  that  empirical  data  were  poor  and  neither  time  nor 
money  was  available  to  collect  more.  They  were  tasked  with  deciding  whether  a 
device  was  required  for  institutional/unit  training  and  whether  the  prototype 
trainer  was  adequate.  Using  analytical  methods,  TRAINVICE  I  (Wheaton  et  al., 

1977)  and  TRAINVICE  II  (Narva,  1979)  were  applied  to  three  candidates.  These 
two  methods  plus  the  questionable  empirical  data  gave  three  identical  rankings 
of  the  candidates  although  the  methods'  results  disagree  on  the  degree  of  differ¬ 
ence.  Therefore,  the  analysts  held  that  this  was  a  reliable  method  of  producing 
a  device  assessment  when  test  data  are  missing  or  inconclusive. 

LaRochelle  (1976)  solved  the  missing  data  problem  differently  in  his  assess¬ 
ment  of  a  System  for  Air-to-Air  Combat  (SAAC)  simulation.  Following  an  analytical 
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examination  of  alternatives  based  on  Osgood's  surface  (qualitative  stimulus** 
response  comparison),  he  concluded  that  positive  transfer  of  training  would  take 
place  but  amount  could  not  be  predicted.  Therefore,  he  postulated  a  series  of 
transfer  from  20%  to  100%.  When  these  percentages  were  applied  to  the  number  of 
flying  missions  in  an  F-15  replaced,  each  set  was  costed  on  a  set  of  relevant 
costs  for  ten  years.  A  further  sensitivity  analysis  .<>3  the  construction  of 
graphs  showing  costs  if  SAAC  and/or  F-15  costs  varied  by  20%  from  the  estimated 
costs. 

The  conclusion  drawn  from  these  types  of  reports  is  that  a  wealth  of  data 
and  patterns  for  conduct  of  TEA  and  CTEA  exists.  The  most  relevant  to  a  new 
study  are  those  conducted  on  similar  or  replacement  systems.  The  reports  show 
that  robustness  of  data  and  reliability  of  conclusions  follow  a  pattern.  The 
most  reliable  are  found  in  such  assessments  as  WSTEA.  These  deal  with  opera¬ 
tional  systems  and  have  access  to  empirical  performance  data  and  judgmental 
evaluations  of  personnel  with  extensive  experience  operating  and  maintaining 
the  system  over  time.  The  assessments  of  developing  systems  are  initially 
analytical,  the  result  of  recourse  to  subject  matter  experts,  theory,  and 
comparison  to  analogous  and  predecessor  systems.  By  the  time  they  are 
fielded,  some  empirical  data  are  available  but  they  are  not  complete  or 
extensive. 

In  the  case  of  training  devices  in  the  developmental  stages,  new  media,  or 
other  training  evaluations,  the  most  reliable  assessments  and  prediction  result 
when  the  system  they  pertain  to  is  already  operational.  Then,  performance 
measures  exist  to  which  the  new  development  may  be  compared. 

The  most  difficult  analysis  is  that  of  a  training  device  for  a  developing 
system.  Regulations  require  that  a  device  be  ready  for  testing  during  system 
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DT/OT  II.  Since  the  device  need  may  not  be  recognized  early,  serious  compres¬ 
sions  of  time  hampers  device  development  and  evaluation. 

2.6  Audio-Visual  Training  Extension  Course  Lessons  Literature  Review 

2.6.1  Background.  Since  1972,  the  Army  has  developed,  produced,  and 
distributed  to  units  Instructional  material  as  part  of  the  Training  Extension 
Course  (TEC)  Program.  The  TEC  status  list  (U.S.  Army  Training  Support  Center, 
1978)  includes  job  performance  aids,  audio-visual  lessons,  audio  cassette 
lessons,  and  printed  texts  as  TEC.  However,  the  TEC  audio-visual  lessons  are 
the  most  widely  used. 

These  are  a  series  of  self-paced  instructional  programs  that  combine 
a  super  8mm  projection  system  with  audio  cassettes  on  a  Beseler  CUE/SEE  auto¬ 
tutorial  device.  There  are  currently  over  1,000  of  these  lessons  covering  a 
wide  array  of  topics.  There  are  lessons,  such  as  camouflage,  useful  to  most 
soldiers  and  lessons  on  such  specialized  tasks  as  ocean  cargo  documentation  for 
the  71N,  cargo  movement  specialist.  These  are  the  type  extension  material 
generally  referred  to  when  Army  personnel  speak  of  'TEC  lessons'.  In  less  then 
10  years,  their  use  has  spread  throughout  the  Army. 

Earlier  research  provided  a  strong  theoretical  base.  A  variety  of  studies 
comparing  the  effectiveness  of  training  media  and  methods  suggested  that  self- 
paced,  audio-visual  instructional  devices  are  a  suitable  (and  perhaps  preferr¬ 
ed)  method  of  providing  ancillary  instruction  in  applied  training  contexts 
(Pieper  et  al.,  1972). 

The  theory  has  been  examined  and  tested  by  TEC  and  TEC-related  work  since 
the  inception  of  the  program.  Much  has  been  done  by  the  ARI,  particularly  the 
Field  Unit  at  Fort  Benning,  Georgia.  However,  the  total  body  of  research  is  not 
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extensive.  A  Report  Bibliography  conducted  by  the  Defense  Technical  Informa¬ 
tion  Center  ii\  September  1980  listed  only  14  documents  under  the  search  term 
'Training  Extension  Course'.  These  and  other  related  documents  are  reviewed 
in  subsequent  sections  examining  TEC  effectiveness,  usage,  cost  and  other 
related  issues. 

2.6.2  Effectiveness .  The  adoption  and  promulgation  of  TEC  lessons  are 
based  on  their  proven  effectiveness  when  compared  to  other  military  instruction¬ 
al  media.  A  number  of  studies  have  addressed  this  topic.  Jacobs  and  Hardy 
(1974)  compared  a  sample  of  379  TEC  users  with  a  188-person  control  group  and 
found  that  MOS  test  performance  was  higher  in  units  that  had  TEC  materials 
available  and  reported  a  high  TEC  use. 

Knerr  et  al.  (1975)  studied  the  comparative  effectiveness  of  TEC  informa¬ 
tion  versus  conventional  Army  classroom  training,  evaluating  both  against  a 
baseline  level  of  performance.  Measured  by  hands-on  performance  tests  as  well 
as  Lesson  Administrative  Instruction  (LAI)  pre-  and  post-tests,  the  TEC-trained 
groups  performed  better  than  or  equally  well  as  all  others.  Analysis  of  results 
showed  that  TEC  was  more  effective  than  conventional  instruction  in  training 
persons  of  lower  mental  ability. 

Leonard  et  al.  (1976)  conducted  research  that  proved  that  TEC  was  an 
effective  medium  for  refresher  training.  Using  six  groups  including  an 
untrained  control,  they  administered  TEC  lessons  on  hand  grenade  selection, 
maintenance,  and  use  to  previously  trained  men.  They  found  significantly  higher 
scores  for  all  subjects  receiving  TEC  refresher  training  except  in  one  category. 
This  category  (identification  of  hand  grenade  components)  had  been  poorly 
covered  in  the  TEC  lesson. 
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Superior  training  effectiveness  and  retention  were  shown  by  Holmgren  et 
al.  (1978)  (also  reported  in  Strasel  et  ax.,  1977)  using  performance  measures 
and  tests  designed  for  a  field  evaluation  of  TEC  (Swezey  and  Eaklns,  1977).  A 
sample  of  1320  Reserve  Component  and  Active  Army  soldiers  in  the  combat  arms 
was  used.  The  research  team  found  that  LAI  pre-  and  post-testing  did  not 
appear  to  make  a  substantive  contribution  to  the  learning  process.  TEC 
instruction  showed  a  small  but  distinct  advantage  over  conventional  instruc¬ 
tion,  both  amount  of  initial  learning  and  percent  of  learning  retained  in 
three  of  five  subject  areas.  In  a  fourth  subject,  initial  learning  with  TEC 
was  no  better  but  retention  was  superior. 

However,  the  foregoing  research  does  not  specifically  address  one  unique 
aspect  of  TEC.  It  is  not  intended  to  supplant  conventional  training  but 
to  complement  it.  The  contribution  of  learning  from  TEC  to  the  total  learning 
acquired  from  all  sources  is  not  clearly  understood  or  defined.  This  was 
recognized  by  Sassone  (1978).  In  summarizing  his  literature  review  of  TEC 
training  effectiveness  he  states: 

In  sum,  because  of  TECs  unique  complementary  role,  it  does  not 
fit  the  usual  training  program  mold,  and  analytical  techniques 
currently  available  in  the  training  literature  are  not  completely 
suitable  for  a  TEC  evaluation. .. .it  appears  a  TEC-specific  evalua¬ 
tion  methodology  must  be  developed. 

2.6.3  Utilization.  Although  all  work  to  date  has  shown  TEC  can  be  an 
effctive  training  medium,  its  effectiveness  may  be  impaired  by  misuse  and 
non-use.  Therefore,  a  number  of  studies  have  addressed  the  quantititative  and 
qualitative  factors  related  to  TEC  usage. 

The  earliest  work,  McCluskey  and  Tripp  (1975),  was  hampered  in  evaluating 
the  TEC  program  implementation  since  sufficient  time  had  not  elapsed  to 
permit  an  adequate  evaluation.  Nevertheless,  they  drew  several  conclusions 
noted  in  most  subsequent  studies.  First,  where  TEC  was  available,  desired 
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levels  of  usage  were  not  reached  because  of  lack  of  command  emphasis  and 
dilution  of  emphasis  and  Information  during  transmission  down  the  chain 
of  command.  Second,  they  found  that,  although  positively  viewed  by  personnel, 
the  mechanics  and  scheduling  of  use  for  Individuals  militated  against  frequent 
employment  of  TEC  as  a  self-paced,  individual  learning  medium.  During  the 
authors'  period  of  study  they  documented  an  increased  usage  of  TEC  in  the 
group  mode. 

A  considerably  more  extensive  survey  of  usage  Is  reported  in  Strasel  et  al. 
(19/7).  (These  data  are  reported  also  in  Mays  et  al.,  1977.)  In  answer  to 
the  research  problem,  "Are  TEC  lessons  being  used  by  members  of  the  target 
audience?"  this  team  surveyed  TEC  users,  non-users,  training  officers  and 
NCOs,  and  training  support  personnel  in  CONUS  and  USAREUR.  Opinions  on 
usage,  attitudes,  and  problems  were  gathered  in  both  Active  Army  and  Reserve 
Component  units.  Data  collected  showed  that  about  50Z  of  the  combat  arms 
personnel  (across  all  categories)  have  used  TEC  lessons.  However,  usage  at 
that  time  was  relatively  low — about  .4  to  .5  uses  per  man  month.  TEC  was  used 
in  the  average  battalion  about  350  times  per  month  and  about  4,560  times  per 
month  in  TRADOC  installations  (schools). 

Other  relevant  findings  were  related  to  attitudes  and  command  and  trainer 
emphasis.  Thirty-five  percent  of  the  potential  users  indicated  that  they  had 
never  heard  of  TEC  until  the  day  they  were  answering  the  quest ionnaire. 

Others  said  they  knew  little  about  it  (35Z),  did  not  know  where  the  materials 
were  kept  (29Z),  their  superiors  (28Z)  or  unit  trainers  (21Z)  did  not  direct 
them  to  use  TEC  (based  on  a  sample  of  3,284  soldier-users).  Some  write-in 
responses  were  related  to  use/non-use — needed  lessons  on  additional  topics, 
lacked  time  for  TEC  study  during  duty  hours,  not  interested  in  TEC,  and  found 
lessons  too  simple  and/or  boring. 
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Mays  et  al.  (1979)  reported  on  a  second  phase  of  the  work  described. 
Findings  In  Phase  2  were  essentially  those  of  Phase  1.  Some  highly  pertinent 
findings  were  reported  for  the  first  time.  They  found  that  90%  of  all  record¬ 
ed  use  occurred  in  groups.  Most  of  this  use  was  mandatory  (81,8%)  and  took 
place  during  duty  hours  (96.6%).  Most  users,  trainers,  and  otner  inter¬ 
viewees  preferred  TEC  to  4  of  7  other  training  methods  (lectures,  small  group 
instruction,  training  films,  and  soldier's  manual)  and  felt  it  should  be 
continued. 

In  a  recent  study  the  Defense  Audit  Service*  (DAS)  found  that  TEC  lessons 

were  net  properly  used.  The  DAS  noted  that  TEC  were  automatically  issued  to 

all  Army  units  at  the  battalion  level  and  that  62%  of  the  TEC  lessons  issued 

were  ..ot  used.  Addressing  this  issue,  L1G  R.  A.  Yerks,  Deputy  Chief  cf  Staff 

for  Personnel,  testified  to  the  Subcommittee  on  the  Department  of  Defense, 

U.S.  House  of  Representatives  Committee  on  Appropriations,  that  the  Army  was 

studying  how  to  change  policies  and  procedures  to  imprwe  the  efficiency  of 

the  Audio/Visual  Distribution  System.  He  stated  that: 

Consideration  is  being  given  to  a  combination  push/pull  system. 

That  is  ,  use  of  minimum  initial  distyribution  with  user  demand 
for  additional  copies  of  TEC  materials.  Products  would  continue 
to  be  distributed  directly  to  using  units  from  the  central  dis¬ 
tribution  facility.  However,  the  Training  and  Audio/Visual 
Support  Centers  supporting  these  unit6  would  assume  respon¬ 
sibility  for  accountability,  additional  loans,  maintenance, 
disposition  and  reporting  uBe  of  copies  based  on  input  from 
using  units.  It  is  anticipated  that  these  changes  will  imprwe 
utilization  statistics  such  as  those  developed  by  the  Defense 
Audit  Service,  i.e.  ,  number  of  copies  distributed  compared  to 
the  number  of  recorded  uses  per  copy.* 

The  low  level  of  usage  is  a  factor  that  must  be  combined  with  cost  and 

effectiveness  data  in  any  assessment  of  TEC  efficiency. 


*U.S.  96th  Congress,  2nd  Session,  DOD  Appro,'  lations  for  1981,  pp.  306-307. 
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2.6.4  Cost.  Most  studies  of  the  TEC  Program  refer  to  it  as  a 


cost-effective  instructional  delivery  medium.  The  proof  of  this  is 
not  always  apparent.  TEC  costs  can  be  examined  as  part  of  a  weapon 
system's  training  life  cycle  cost,  as  an  addition  to  individual  KOS 
training  or  costed  for  annual  budgetary  purposes  as  a  unique  program 
produced  and  distributed  from  TRADOC. 

The  COEA  conducted  by  the  Combact  Arms  Training  Board  (CATB) 
(Department  of  Army,  1975)  reported  the  first  TEC  cost  analysis. 

They  found  TEC  more  cost-effective  than  conventional  instruction 
($42.7  mil/$50.5  mil)  based  on  the  same  number  of  instructional  hours. 

Much  of  the  predicted  savings  was  based  on  cost  avoidance.  Tc.C  was 
foreseen  as  requiring  less  officer/iCO  support  at  the  battalion  level. 

The  cost  data  in  the  COEA  were  supplied  by  ARI  (.Tank  in  et  al., 

1975).  Temkin's  report  contains  more  detail  than  CATB's.  Among  their 
points  was  that  TEC  would  become  less  costly  in  comparison  to  conventional 
instruction  only  as  volume  of  usage  increased.  Their  costing  assumed 
optimal  use  of  TEC  by  individuals.  They  did  not  estimate  costs  considering 
group  use  or  under-utilization.  Conventional  group  training  was  the  only 
instructional  alternative  compared  to  TEC.  Looking  into  the  future  they 
foresaw  that  the  Beseler  CUE/SEE  and  tape  cassette  player  would  be  "good 
for  another  13  years"  (i.e.,  1984-35). 

Litton  Mellonics  personnel  at  Fort  Benning  conducted  the  most  com¬ 
plete  investigation  of  TEC  cost  (see  Strasel  et  al.  ,  1977,  Appendix  A, 

Cost  Benefits:  TEC  use  in  TRADOC  Schools  and  Appendix  B,  TEC  Cost  and 
BeiefJ.ts  Quantification).  They  report  that  TEC  program  costs  for  FY  78 
were  estimated  at  $22.5  million,  about  1%  of  the  total  Army  $2,116  million 
Program  IT  or  training  budget. 
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Tills  study  is  the  first  to  note  that  there  are  two  cost  effectiveness 
nodels  that  can  be  applied  to  TEC  as  follows: 

TEC  COST  EFFECTIVENESS  MODELS 

SUPPLEMENTAL  MODELS: 

-  TEC  is  viewed  as  a  supplement  to  conventional  training 
Costs  are  supplemental  to  base  costs 

Decision  to  continue  based  on  value  placed  on  relative 
effectiveness  to  cost  of  base  and  supplement  (i.e. , 
increased  value  of  effectiveness  to  cost  for  the 
supplement) 

REPLACEMENT  MODELS: 

-  TEC  is  viewed  as  a  replacement  for  conventional  training 

-  Cost  savings  may  accrue 

-  Decision  to  continue  based  on  direct  comparison  of  ratio 
effectiveness  to  cost  for  alternate  cases  (i.e.,  cost 
savings  may  be  realized) 

First,  the  investigation  of  TEC  program  costs  developed  sunk  costs 
(money  spent  or  obligated  before  FY  78  -  FY  37),  Displayed  in  many 
different  ways  and  categories  these  analyses  are  a  model  of  exhaustive 
cost  determination  and  prediction.  From  these  analyses  they  determined 
that  each  copy  of  a  lesson  provided  to  a  battalion  costs  the  Array  $34. 

Second,  the  cost  avoidance  to  the  Army  resulting  from  use  of  already 
existing  TEC  materials  by  TRADOC  Branch  Schools  to  replace  or  supplement 
conventional  instruction  was  calculated.  Schools  reported  savings  of 
over  $104,000,  cost  avoidance  of  over  $2  million  in  FY  79  due  to  TEC  use 
and  estimated  $3  million  potential  costs  not  spent  to  develop  new  remedial 
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instruction  since  TEC  existing  material  was  used. 

A  third  cost  investigation  quantified  the  benefits  that  •'ccrued 
when  TEC  was  used.  TEC  soldiers  achieved  proficiency  for  less  training 
resources  (time  and  ammunition).  On  selected  lessons  they  found  an 
average  9.4%  gain  in  proficiency  of  an  individual  skill  for  $0.39 
per  person. 

The  study  recommended  an  extension  of  the  work,  from  individual 
combat  arras  soldier  to  unit.  To  this  end,  Sassone  and  Bercos  (1979) 
developed  a  methodology  for  determining  the  value  of  TEC  in  terms  of 
unit  performance  on  the  ARTLP  evaluation.  The  methodology,  thus  far 
exercised  only  with  hypothetical  data,  is  an  adaption  of  economic 
welfare  models. 

The  DAS  Report  cited  earlier  estimated  that: 

"$26.1  million  could  be  saved  ‘a  reproduction  and  distri¬ 
bution  materials  if  the  Army's  training  and  audio-visual 
support  center  at  each  installation  controlled  and  issued 
TEC  lessons  to  troop  units  as  needed  instead  of  automati¬ 
cally  distributing  the  lessons,"* 

LTC  Robert  A.  Yerks,  Deputy  Chief  of  Staff  for  Personnel,  in  testimony 
referred  to  above,*  stated  that  development  for  new  lessons  had  been 
slowed  and  curtailed  as  well  as  replacing  the  audio-visual  cassette 
lessons  with  print  media  where  possible.  lie  held  that  considerable 
but  unquantified  cost  saving  would  accrue  in  this  program. 

2.6.5  Other  Issues.  This  literature  review  uncovered  other  TEC- 
related  topics  in  the  material  analyzed.  Proposed  management  and  selection 
systems  for  TEC  —  both  as  an  audio  visual  medium  and  one  of  many  exportable 


*Ibid 
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extension  media  —  have  been  developed  (Taylor  et  al. ,  1977;  Bialek 
et  al. ,  1978;  Butler  et  al.  ,  1973).  The  trend  seems  to  be  toward 
considering  TEC  audio  visual  lessons  as  only  one  of  an  array  of  in¬ 
structional  media  to  be  selected  when  its  strengths  —  use  by  group 
or  individual,  self-paced,  some  feedback,  exportable  to  a  wide  variety 
of  learning  contexts  —  make  it  an  efficient  choice. 

Indermill  and  Reeder  (1975)  perceived  a  similarity  bttween  TEC 
lessons  and  self-study  correspondence  course  lessons.  They  have  de¬ 
veloped  a  manual  to  teach  training  developers  to  convert  TEC  lessons 
to  correspondence  lessons,  effecting  a  considerable  savings  in  original 
development  costs.  Another  effort  aimed  at  reducing  cost  was  Markham's 
(1979)  v/ork  in  achievement  as  related  to  aesthetic  embellishment  in 
education  art.  Using  TEC  visuals  and  simple  line  drawings  to  illustrate 
the  same  twenty  items,  he  found  no  significant  difference  in  learning  as 
measured  by  an  end  of  course  test.  He  estimated  that  a  complex  art  frame 
for  TEC  costs  $75  while  a  simple  one  costs  only  $10. 

Of  great  interest  are  those  studies  that  look  to  the  future.  Al¬ 
though  the  TEC  Program  is  not  yet  10  years  old,  a  replacement  is  already 
on  the  scene.  Hawkins  and  ICribs  (1979)  report  that  TRADOC  is  investigating 
the  training  and  cost-effectiveness  of  converting  its  TEC  materials  from 
the  Beseler  CUE/SEE  to  videodisc.  Preliminary  work  is  being  conducted 
at  Fort  Sam  Houston.  In  work  for  the  U.S.  Army  Field  Artillery  School 
(USAFAS)  at  Fort  Sill,  Hoyt  and  his  associates  (1977)  have  determined 
that  TEC  lessons  can  readily  be  converted  to  computer-assisted  instruction 
(CA1)  using  a  "formal  ISD  process."  They  next  taught  USAFAS  course 
development  personnel  to  do  this  at  a  very  reasonable  expenditure  of 
time  and  money.  They  foresee  this  as  low  cost  exportable  training  since 
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expensive  dedicated  CAI  systems  may  not  be  necessary.  At  least  for  the 
Field  Artillery,  deployed  time-shared,  mini,  and/or  tactical  computers  may  be 
used.  Hawkins  and  Kribs  also  find  the  emerging  video  and  computer-produced 
lessons  more  cost  effective  by  25-75%.  These  are  savings  in  developmental 
costs  since  texts,  graphics,  etc.  may  be  created  by  and  stored  in  the  computer. 
These  types  are  predicted  to  be  more  effective  because  of  superior  individual¬ 
ization  and  feedback  capability. 

Holmgren  et  al.  (1979)  have  undertaken  initial  work  comparing  the  effec¬ 
tiveness  of  TEC  lessons  on  a  prototype  videodisc  player  with  the  audio  visuals 
on  the  Beseler  CUE/SEE.  They  found  that  trainees  who  received  either  type  of 
supplementary  training  did  better  than  a  control  group  who  received  neither. 

The  difference  between  the  two  groups  with  augmented  training  was  not  found 
statistically  significant  for  two  sets  of  lessons.  On  the  third  set,  the 
CUE/SEE  trained  soldiers  had  a  significantly  higher  performance  test  score 
than  the  videodisc  trained  soldiers. 

2.6.6  Abstracts .  Appendix  A  contains  abstracts  of  31  documents  relevant 
to  a  study  of  TEC.  In  some  cases,  there  are  more  substantial  condensations 
of  the  literature  items  with  charts,  tables,  or  other  graphics  as  deemed 
especially  illustrative.  The  abstracts  are  arranged  in  chronological  order 
from  June  1974  to  the  present  date,  enabling  the  reader  to  trace  the  initia¬ 
tion  and  development  of  the  TEC  programs. 

2  •  ^  Summary  of  Review  Findings 

The  literature  reviewed  above  contains  many  examples  of  CTEA  applications 
and  many  different  models  and  methods  were  used  in  these  applications.  Some 
of  the  applications  (and  methods)  were  neither  cost  analyses  nor  training 
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effectiveness  analyses*  Some  treat  only  a  specific  issue  concerning  a  weapon 
system,  while  some  consider  multiple  aspects  and  multiple  factors  related 
to  training  systems  and  their  relative  cost  effectiveness.  None  of  these 
approaches  pretends  to  be  a  C1EA  methodology  that  is  generalizable  to  all  Army 
training  systems. 

More  importantly,  for  this  report,  none  appears  sufficiently  broad  in 
scope  and  approach  to  be  developed  into  a  generalizable  methodology.  For 
example,  none  of  these  methods  provide  all  of  the  information  required  for 
decisions  at  all  points  in  the  LCSMM.  However,  detailed  examination  of  the 
techniques,  steps,  and  processes  ::ithin  the  methods  (the  process  elements  from 
which  the  methods  are  built)  showed  that  each  method  could  contribute  to  the 
CTEA  job  to  be  performed  at  some  point  in  the  LCSMM.  Some  methods  contribute 
more  information  elements  within  some  data  situations  than  do  others.  For 
example  informal  expert  judgment  methods  are  useful  in  generating  task  lists 
when  none  are  available.  However,  when  task  lists  are  available  or  have  been 
generated,  the  formal  analytic  models  for  prediction  of  training  programs  can 
provide  superior  CTEA  outputs. 

The  CTEA  methods  reviewed  above  can  be  systematically  combined  and/or 
ordered  to  provide  a  framework  for  directing  an  analyst  when  to  use  any  or  a 
combination  of  these  methods  for  a  particular  problem  type.  Such  a  systematic 
approach  or  framework  for  CTEA  applications  is  discussed  in  the  next  section. 
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A  SYSTEMATIC  APPROACH  TO  CTEA  FOR  VARIED  SYSTEM  SITUATIONS 

3 . 1  Introduction 

The  Army's  objective  in  the  area  of  training  must  be  to  provide  soldiers 
and  units  effective  training  in  a  time  efficient  manner  and  at  the  least 
possible  cost.  Effectiveness  is  attained  when  training  objectives  and  job 
performance  standards  are  met.  Effectiveness  can  usually  be  increased  through 
careful  selection  of  what  is  to  be  trained,  where  it  is  to  be  trained,  and  how 
it  is  to  be  trained;  by  evaluating  the  training  and  job  performance;  and 
revising  training  until  it  meets  the  objectives.  Training  must  be  effective 
within  the  time  available  for  it;  thus,  effectiveness  and  time  efficiency  must 
be  examined  together.  Finally,  effective  training  must  be  delivered  in  the 
available  time  without  excessive  cost.  These  principles  imply  the  structuring 
and  evaluation  of  alternative  training  programs  either  for  competing  develop¬ 
ing  systems  or  for  a  single  fielded  system. 

Evaluation  of  training,  taking  into  consideration  effectiveness,  effi¬ 
ciency,  and  cost,  is  a  continuous  process  and  is  not  dependent  on  actuality  of 
ongoing  training  or  performance  measures.  Proposed,  innovative,  and  unique 
training  systems  and  media  also  require  assessment. 

As  described  in  the  review  of  Section  2,  techniques  and  processes  for 
accomplishing  the  required  cost  and  training  effectiveness  analyses  and 
evaluations  are  available  but  have  not  been  systematized  into  an  integrated 
family  of  models  suitable  for  use  by  the  training  developers  and  training 
analysts.  Such  systematization  is  required  training  analysts  are  to  design 
and  carry  out  an  evaluation  that  considers  all  contingencies. 

A  systematic  approach  to  training  system  CTEA  is  presented  below. 

This  includes  an  overview  model  of  CTEA  that  is  generalizable  to  nonsystem 
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training  and  fielded  systems  as  well  as  covering  developing  systems.  This 
systematic  approach  defines  the  analytic  processes  that  the  analyst  must 
consider  depending  on  the  response  to  specific  questions.  The  systematic 
approach  to  CTEA  identifies  and  relates  a  family  of  study  categories.  These 
are  CTEA  (TRADOC  Reg.  350-h),  the  ISTEA,  and  TDS  (device-related)  for  devel¬ 
oping  systems;  and  the  ISTEA,  TSEA,  TDS,  and  Training  Evaluation  for  fielded 
systems.  Table  3-1  shows  the  relationships  of  subcategories  by  type. 

The  approach  leads  to  identification  of  the  appropriate  submodels  to  be 
applied.  This  then  increases  the  precision  of  the  questions  and  defines 
specific  processes  required. 

The  following  sections  present  and  describe  the  systematic  approach  to 
training  system  analysis  and  the  submodels  as  specific  components  of  the 
general  CTEA  approach. 

3 . 2  The  Systematic  Approach  to  Training  System  Analysis  and  CTEA 

Training  analysis,  as  noted  above,  is  a  comprehensive  process  that  is 
carried  out  for  a  number  of  different  purposes.  The  Army  has  identified  and 
defined  a  number  of  these  as  they  relate  to  operational  systems  and  training 
devices.  This  identification  approach  allows  the  analyst  systematically  to 
pinpoint  an  assessment  category  whose  study  approach  is  then  described  in  the 
appropriate  submodel.  The  submodels  can  be  considered  to  form  a  general  model 
of  training  system  analysis. 

The  systematic  approach  classifies  the  assessment  needs  on  the  basis 
of  the  answers  to  a  set  of  very  general  training  questions.  The  regulations 
do  not  prescribe  analysis  models  for  the  evaluation  of  nonsystem  training. 
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Table  3-1.  Categories  of  Assessment  Types  Within  the  Generic  CTEA 


DEVELOPING 

FIELDED 

s 

CTEA  CTRADOC  REG.  350-4) 

ISTEA 

Y 

TDS  (a ) 

IS TEA (a) 

S 

TSEA 

T 

TDS 

E 

TDS (a) 

M 

N 

0 

N 

S 

TDS (a) 

TDS 

Y 

COE A/ CTEA* 

TRAINING  EVALUATION 

S 

T 

E 

M 

*The  CTEA  for  a  major  nonsystem  training  device  or  simulation  is  in 
fact  a  CGEA  for  that  training  device  or  simulation  (e.g.f  MILES). 


Taking  the  varieties  of  training  analysis  into  consideration,  a  general  model 
of  training  system  analysis  has  been  developed  (Figure  3-1).  This  model 
separates  all  the  currently  identified  common  types  of  analysis.  Using  a 
logic  approach,  the  analyst  can  identify  the  type  training  assessment  required. 

The  analysis  flow  is  as  follows.  The  first  discriminator  is  whether 
new  training  is  to  be  developed.  If  the  answer  is  yes,  then  the  analyst 
determines  if  it  is  system  or  nonsystem  training.  If  system  training,  he 
is  directed  to  use  the  CTEA  (LCSMM)  and  the  ISD  submodels.  Nonsystem  training 
also  requires  ISD  for  design  but  a  different  submodel,  the  Training  Evaluation, 
for  assessment.  If  the  initial  question's  answer  indicated  that  the  training 
to  be  analyzed  already  exists,  the  analyst  again  is  directed  along  alternate 
paths  for  system  and  nonsystem  training.  Nonsystem  training  is  assessed  using 
the  Training  Evaluation  submodel.  (It  has  been  designed  for  use  with  either 
developing  or  established  nonsystem  training.)  The  ISTEA  is  the  submodel  used 
to  answer  cost,  effectiveness,  or  efficiency  questions  pertaining  to  system 
training.  Should  the  ISTEA  uncover  a  performance  gap,  the  analyst  is  directed 
to  the  TSEA.  The  TSEA  further  identifies  the  cause  of  this  gap.  If  it  is 
not  due  to  training  (i.e.,  due  to  hardware,  personnel,  or  logistic  support 
subsystems),  a  TSE  is  called  for.  If  the  training  subsystem  needs  a  fix, 
a  TDS  is  conducted.  This  leaves  only  major  device/simulation  questions 
unanswered.  If  they  have  been  raised  in  the  earlier  analyses,  a  Device 
Effectiveness  Analysis  following  a  device-oriented  TDS  completes  the  evalu¬ 
ation.  If  other  substantive,  unique  analysis  types  are  uncovered,  the  model 
is  flexible  and  can  be  enlarged  to  accommodate  them. 

The  submodels  shown  in  Figure  3-1  are  designated  by  off-page  exit  numbers. 
These  numbers  are  the  entrance  symbols  in  the  succeeding  figures  depicting 


submodel  logic  flow. 
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3.3  CTEA  (TRADOC  Reg.  330-4)  Model 


The  CTEA  model  (Figure  3-2)  is  a  fully  developed  method  with  suggested 
processes  to  carry  out  all  required  analysis  identified  (Matlick  et  al., 
1980a).  By  definition  the  CTEA  is  related  to  an  operational  system  and 
requires  comparison  of  alternatives  as  to  cost-effectiveness  (C/E)  and  non- 
quantifiable  issues.  The  constraints  to  successful  analysis  lie  in  the 
availability  of  data.  The  choice  of  methods  results  from  the  analyst's 
response  to  questions  covering  the  existence  of  (1)  a  task  list,  (2)  a 
training  program,  (3)  training  program  alternatives,  and  (4)  comparable 
effectiveness  data  for  all  alternatives.  Processes  are  suggested  for  gener¬ 
ating  task  lists,  predicting  training  program  alternatives,  estimating 
effectiveness,  costing,  making  C/E  analyses,  and  resolving  issues.  This  very 
adaptable  model  enables  the  analyst  (within  the  availability  of  data)  to 
conduct  a  CTEA  and  produce  reliable  recommendations  at  any  stage  of  the 
LCSMM. 

Litton  Mellonics  has  prepared  a  guidebook  (Matlick  et  al.  ,  1980a,  p.  v2) 
demonstrating  the  CTEA  model  application  for  the  most  common  data/no  data 
situations.  (A  description  of  the  model  may  be  found  in  Section  2.) 

3.4  ISP 

The  ISD  model  is  a  set  of  procedures  for  developing  and  monitoring  the 
effectiveness  of  training  systems  (see  Section  2  for  a  brief  description  of 
ISO).  Phases  I  -  III  are  developmental  processes,  and  Phase  V  Is  a  training 
assessment  model.  ISD  relates  to  the  systematic  approach  to  TEA  (as  submodel 
2)  as  a  training  development  process.  The  general  model  directs  the  analysis 
to  ISD  if  new  system  or  nonsystem  training  development  is  needed.  It  is 
highly  likely  that  when  processes  are  identified  for  TDS,  ISD  (or  ISD 
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Figure  3-2.  CTEA  (TRADOC  REG  350-4)  MODEL  FOR  DEVELOPING  SYSTEMS 


abbreviated  for  media/method  selection)  will  be  one  of  the  processes  of 
choice.  Since  ISD  is  a  continuous  process,  future  work  will  identify  the 
paths  from  ISD  to  the  TEA  submodels  for  appropriate  analysis  of  training 
following  implementation.  An  ISD  procedures  overview  is  shown  in  Figure  3-3. 

3.5  The  Training  Evaluation  Model 

In  order  to  complete  the  family  of  submodels  and  provide  for  assessment 
of  all  major  training  types,  Litton  has  developed  a  model  for  the  evaluation 
of  nonsystems  (Figure  3-4).  The  proposed  model  is  designed  to  evaluate  either 
existing  nonsystem  training,  if  available,  or  to  evaluate  developing  nonsystem 
training.  The  requisite  model  processes  have  not  been  evaluated  nor  has  the 
model  been  exercised.  Unlike  the  other  submodels,  the  training  evaluation 
model  is  a  new  approach.  Therefore,  the  rationale  underlying  the  development 
follows. 

Problems  exist  in  approaching  evaluation  of  nonsystem  training.  Training 
whose  purpose  is  not  directly  related  to  an  operational  system  may  not  result 
in  observable,  measurable  performance.  Training  in  military  courtesy,  for 
example,  may  result  in  measures  of  knowledge  (written  or  oral  test  scores)  or 
perhaps,  even  performances  under  controlled  or  artificial  conditions.  However, 
the  analyst  will  not  know  if  training  has  resulted  in  appropriate  soldier 
behavior  in  regards  to  military  courtesy  in  the  real  world.  On  the  other 
hand,  other  nonsystem  skills  such  as  map  reading  can  be  tested  in  field 
exercises  and  task  skill  performance  measured.  However,  such  observations  or 
measurements  still  leave  some  questions  unanswered.  For  example,  could  the 
soldier  have  successfully  carried  out  the  same  exercise  under  other  conditions 
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Figure  3-3.  Overview  of  ISD  Procedures 


Fi^uro  3-V  1  of 


of  terrain,  climate,  and  weather?  Would  he  have  been  successful  under  combat 
conditions?  Could  he  solve  a  navigation  problem  not  posed  in  this  exercise? 
Sophisticated  simulation  that  could,  of  course,  provide  reasonable  answers  to 
such  questions  is  not  generally  available  to  a  training  systen  analyst.  The 
analyst  alternatively  can  turn  to  an  examination  of  training  media  and  methods 
to  determine  if  research  and  experience  show  them  to  be  valid  means  for 
producing  the  required  results. 

The  central  question  of  the  latter  approach  is:  Given  these  required 
results,  are  these  methods  and  media  valid?  Hence,  knowledge  of  the  effects 
of  training  rather  than  knowledge  of  results  is  pivotal.  That  is,  an  analyst's 
knowledge  of  the  proven  effects  of  methods  and  media  can  be  used  to  predict 
their  effectiveness  in  new  contexts.  For  example,  he  may  know  that  a  certain 
method  has  the  effect  of  improving  the  trainee's  ability  to  generalize  prin¬ 
ciples  to  a  large  variety  of  specific  situations.  He  might  then  predict  that 
this  method  is  a  valid  way  to  teach  map  reading.  Likewise,  che  analyst  may 
know  that  the  formal  statement  of  training  objectives  clarifies  the  intended 
outcomes  for  both  student  and  trainer  or  that  the  development  of  valxd  and 
reliable  instruments  for  the  measurement  of  learning  establishes  a  basis  for 
feedback  to  both  the  learner  and  the  instructional  system.  This  knowledge 
gives  the  analyst  assurance  that  he  can  predict  reliable  means  of  enhancing 
desired  effects. 

Such  an  analytical  process  tends  to  mirror  the  ISD  procedures  —  the 
analytical  processes  that  create  new  training  programs.  The  analytical 
approach  to  training  evaluation  described  above  could,  in  effect,  provide  a 
quality  control  check  on  ISD  processes  as  well  as  refine  or  correct  their 
output.  Note  that  this  approach  is  distinctly  different  from  ISD  Phase  V, 
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Control.  ISD  Phase  V  Is  based  on  the  feedback  of  performance  and  effective¬ 
ness  data  to  training  program  proponents.  In  a  flawed  program,  these  data  may 
be  in  error  or  in  need  of  revision. 

The  Training  Evaluation  Model  (Figure  3-4)  depicts  this  analytical 
approach  to  training.  According  to  this  model,  training  evaluation  begins 
with  indications  of  some  sort  that  performance  is  not  adequate  and/or  is 
excessively  costly.  In  the  case  of  developing  nonsystem  training,  the  impetus 
for  the  evaluation  will  probably  be  to  compare  posed  alternatives  or  to 
compare  an  alternative  to  a  current  baseline,  although  need  for  a  "fix"  might 
be  recognized  before  the  nonsystem  training  is  fully  developed  and  fielded. 
These  indications  may  be  formal  or  informal,  many  or  few,  depending  on  the 
importance  attached  to  the  training  program  in  question.  They  must  be 
examined  closely  for  validity  early  in  the  evaluation. 

If  the  job  performance  measures  are  valid  or  if  there  are  other  per¬ 
suasive  indications  of  inadequate  performance,  the  evaluation  proceeds  to 
front-end  analysis  to  more  closely  define  the  problem  or  pinpoint  problem 
locations.  If  no  performance  inadequacies  are  detected,  cost  problems  are 
examined.  The  front-end  analysis  may  show  that  what  initially  appears  to  be 
a  training  problem  is  actually  some  other  kind  of  problem  (e.g.,  an  environ¬ 
mental  or  psychological  problem  that  has  adversely  affected  performance).  If 
there  is  a  real  training  requirement,  the  evaluation  proceeds  to  job/ task 
analysis. 

The  training  developers  normally  conduct  job/task  analysis  for  existing 
or  new  training  programs.  If  this  has  not  been  done,  the  analyst  must  use  an 
appropriate  process  for  preparing  a  provisional  critical  job/task  list  and 
analysis.  The  purpose  of  the  job/ task  analysis  within  the  Training  Evaluation 
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Model  is  to  determine  if  the  essential  tasks/skills  have  been  selected  for 
training.  If  they  have  not,  the  analysis  must  cease,  and  an  ISD  process 
undertaken  to  develop  appropriate  training.  If,  on  the  other  hand,  the 
critical  tasks  have  been  correctly  identified  and  selected  for  training,  the 
evaluation  moves  on  to  an  examination  of  how  they  have  been  or  are  to  be 
trained. 

If  there  are  data  on  instructional  effectiveness  (such  as  written,  oral, 
or  performance  test  scores),  their  validity  is  analyzed  unless  it  has  already 
been  established.  If  the  data  are  not  valid,  the  analyst  must  again  revert  to 
the  ISD  model  to  develop  valid  tests/measures/data  or  training.  If  the  data 
are  valid,  they  become  the  basis  for  deciding  if  instruction  is  effective 
(that  is,  if  what  is  being  taught  is  actually  being  learned).  If  instruction 
is  not  found  effective,  the  evaluation  proceeds  to  the  prediction  of  other 
methods,  media,  or  training  elements  expected  to  achieve  effective  instruction. 
This  process  results  in  a  theoretical  instructional  system.  The  continuation 
of  the  evaluation  is  not  dependent  upon  existence  of  real  world  Instructional 
data.  Their  absence  is  not  the  same  problem  as  that  of  invalid  data.  In  the 
case  of  predicted  programs,  the  analyst  may  employ  judgmental  and  analytical 
methods  for  forecasting  effectiveness.  In  the  case  of  operational  training 
systems,  quite  valid  informal  or  intuitive  feedback  may  be  present  and  may,  in 
fact,  be  discovered  as  the  evaluation  proceeds.  Furthermore,  the  absence  of 
such  data  pinpoints  another  deficient  area.  Such  a  deficiency  is  due  either 
to  a  lack  of  formal  instruments  or  means  for  assessing  instructional  effec¬ 
tiveness  or  to  nonuse  of  available  means  or  instruments.  Therefore,  sub¬ 
sequent  evaluation  would  have  a  legitimate  interest  in  discovering  the  reasons 
for  such  lack  or  nonuse  and  in  recommending  the  development  and  use  of  valid 


instruments  and  means. 


On  the  basis  of  comparison  of  alternative  instructional  systems  (theo¬ 
retical  and/or  existing),  a  judgment  is  made  about  the  predicted  effectiveness 
of  the  methods,  media  and  other  elements  of  the  systems.  If  the  empirical 
data  or  the  analytical  processes  indicate  that  the  instruction  is  (or  is 
expected  to  be)  effective,  the  evaluation  proceeds  to  search  for  some  other 
explanation  for  inadequate  performance.  Alternatively,  investigation  will 
search  for  an  explanation  of  why  empirical  data  do  not  demonstrate  effective 
instruction  even  though  analysis  has  indicated  that  methods,  media,  and  other 
elements  are  appropriate. 

This  search  is  accomplished  through  an  analysis  of  potential  or  actual 
transfer  of  training  (TOT)  and  of  skill  retention/decay.  Very  little  of  the 
learning  may  be  transferring  to  the  job  or  there  may  be  interference  within 
the  instructional  system.  If  the  resultant  finding  is  that  TOT  or  retention 
is  a  problem  source,  then  the  analyst  decides  if  adequate  knowledge  exists 
that  will  permit  revision  of  instruction  to  optimize  for  these  factors.  If 
such  knowledge  exists  (or  can  be  readily  and  economically  acquired),  the 
evaluation  proceeds  to  analysis  or  study  of  such  optimizing  designs. 

Whether  or  not  optimization  for  TOT  or  retention  is  undertaken,  the  final 
area  for  analysis  is  cost.  Following  a  cost  analysis,  a  decision  is  made  as 
to  whether  actual  or  projected  costs  are  justified  by  the  actual  or  predicted 
outcomes.  If  the  answer  is  positive,  the  training  evaluation  is  complete.  If 
not,  instructional  system  elements  are  optimized  for  cost  (a  cost-effectiveness 
(C/E)  process)  as  the  last  evaluation  step. 

If  following  the  prediction  of  effective  training,  the  training  system 
elements  are  not  found  appropriate,  the  evaluation  proceeds  to  a  consideration 
of  how  they  may  be  improved.  The  analyst  employs  a  process  to  determine  the 
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system  revision  or  alternation  required  to  prwide  effective  training.  This 
optimization-of-training  process  also  leads  to  cost  and  C/E  analysis. 

The  key  to  the  validation  of  this  model  for  nonsystem  training  evaluation 
is  the  discovery  or  development  of  methods  for  the  execution  of  each  embedded 
process  (shown  in  Figure  3-4  as  rectangles).  This  is  beyond  the  scope  of  this 
project,  but  see  Matlick  et  al.  ,  1980a  and  b  for  the  methods  selected  for  each 
process  in  the  CTEA  model  (Figure  3-2). 

3.6  1STEA 

The  Initial  Screening  Training  Effectiveness  Analysis  (ISTEA)  is  an 
assessment  of  fielded  hardware  system  related  training.  Its  primary  objective 
is  to  determine  if  a  significant  performance  gap  exists  between  designed 
equipment  capability  and  actual  performance  achieved  in  operation  of  a  given 
hardware  system.  Second,  the  ISTEA  is  conducted  to  prwide  data  on  the 
factors  that  affect  performance.  This  knowledge  determines  the  need  for 
further  study  and  the  type  of  future  study  required. 

ISTEA  are  conducted  in  response  to  either  of  two  requirements: 

(1)  to  fulfill  the  requirement  for  continued  quality  control  assessment 
of  ongoing  training; 

(2)  to  prcvide  a  baseline  of  data  on  a  fielded  system  to  be  used  in  the 
CTEA  of  an  analogous  developing  system. 

In  both  cases  analysis  proceeds  by  answering  the  same  set  of  three  consecutive 
questions . 

3.6.1  ISTEA  -  Objective  1.  ISTEA,  Objective  1,  (see  Figure  3-5)  are 
conducted  to  verify  indications  that  actual  system  effectiveness  (E  )  has 

A 

fallen  below  the  presumed  or  formerly-established  system  capability.  The 
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first  question  directing  the  course  of  the  ISTEA  is  whether  the  design  effec¬ 
tiveness  (Ep)  is  known.  If  rot,  the  analyst  must  procure  the  best  empirical 
data  available  on  system  effectiveness.  Some  suggested  data  sources  are 
DARCOM,  the  manufacturer,  MASAA,  OTEA,  reports  of  COEA,  CTEA,  and  field 
experiences.  If  the  valid  empirical  performance  data  are  not  available,  the 
ISTEA  cannot  continue.  Unlike  CTEA  and  other  types  of  comparison,  real-world 
data  rather  than  estimates  are  required  for  the  ISTEA.  Without  them  it  is  not 
possible  to  assert  that  a  performance  gap  exists  or  to  measure  its  size 
accurately . 

If  the  Ep  is  known  or  has  been  determined,  the  assessment  moves  to  the 
second  question.  The  analyst  ascertains  if  E  is  known.  When  it  is  not,  he 

A 

employs  processes  to  measure  the  E  ,  including  as  required,  a  test  of  soldier 

A 

proficiency . 

When  the  ISTEA  analyst  has  measures  of  both  E_  and  E  ,  he  can  deter- 

U  A 

mine  if  design  effectiveness  has  been  substantially  achieved  and  if  not,  the 
size  of  the  performance  gap.  There  is  no  further  analytical  requirement  if 
E^  is  approximately  equal  to  Ep.  When  the  ISTEA  finding  is  that  E^  is 
significantly  less  than  Ep ,  it  is  recommended  that  a  TSEA  be  conducted  to 
locate  the  reasons  for  deficiency. 

3.6.2  ISTEA  -  Objective  2.  An  ISTEA,  Objective  2,  (see  Figure  3-6)  is 
conducted  to  provide  a  comparative  baseline  of  data  from  an  operating  system 
for  use  in  the  CTEA  of  a  developing  system.  As  such,  as  much  detailed  infor¬ 
mation  as  possible  is  gathered  on  the  system  affective  factors  (see  Table 
3-2). 
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Table  3-2  Data  Elements  Collected  for  ISTEA,  Objective  2 


Soldier 

demographic 
attitudes 
learning  styles 
capabilities 

Trainer 

demographics 

attitudes 

effectiveness 

Training  Subsystem 

Hardware  Subsystem 

Training  Environment 
management 
conduct 
priority 

resource  allocation 
trainer  qualifications 
training  records 
command  eiwironment 
personnel 

geography ,  meteorology 


As  in  ISTEA,  Objective  1,  the  first  requirement  is  knowledge  of  the  E^. 

Next ,  the  analyst  obtains  a  known  or  predict  E^.  In  the  process  of  E^ 
determination,  the  analyst  acquires  detailed  data  on  the  soldier,  the  trainer, 
the  training  and  hardware  subsystems,  and  the  training  environment. 

Following  determination  of  E^  and  E^,  the  existence  of  the  perfor¬ 
mance  gap  is  verified  and  the  size  measured.  For  the  purpose  of  the  CTEA,  if 
E  is  less  than  E  ,  training  is  redesigned,  personnel  selection  criteria 
are  revised,  and  hardware  changes  recommended,  depending  on  the  reasons  found 
for  training  system  deficiency.  All  data  and  their  assessment  form  the  input 
provided  to  the  CTEA  study  team. 
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3.7  TSEA 


The  Training  Subsystem  Effectiveness  Analysis  (TSEA)  (sec  Figure  3-7)  Is 
the  most  rigorous  investigation  conducted  on  the  training  related  to  a  fielded 
hardware  system.  It  may  result  from  the  findings  of  an  ISTEA,  Objective  1 
type,  or  if  obv  ious ,  valid  evidence  of  a  significant  performance  deficiency 
exists,  TRADOC  Reg.  350-4  recommends  that  a  TSEA  be  conducted  in  lieu  of  an 
ISTEA.  When  an  ISTEA  has  not  been  done,  the  first  task  of  the  TSEA  analyst  is 
to  collect  and  analyze  those  data  eJ events  usually  collected  in  an  ISTEA  (see 
Table  3-2).  If  the  TSEA  is  an  ISTEA  follow-on,  the  existing  data  are  reviewed. 
If  data  gaps  exist  or  if  some  data  are  outdated  ,  these  deficiencies  are 
compensated  for  by  a  new  data  collection  and  analyses.  These  detailed  analyses 
of  the  training,  hardware,  and  personnel  subsystems  as  well  as  the  training/ 
working  environment  lead  to  pinpointing  of  major  problem  areas  and  substantial 
contributory  factors. 

The  findings  and  recommendations  of  a  TSEA  may  identify  the  problem  locus 
in  any  subsystem  (training  hardware,  logistic  support,  or  personnel)  or  in  any 
combination  of  subsystems.  If  the  training  subsystem  alone  is  implicated,  a 
Training  Development  Study  (TDS)  follows  the  TSEA  to  develop  a  'fix'.  If 
other  subsystems  are  involved,  a  Total  System  Evaluation  to  identify  and 
recommend  'fixes'  for  these  interactive  subsystems  is  indicated. 

3.8  TDS 

3.8.1  General.  When  a  training  subsystem  has  been  found  deficient 
(e.g.  ,  as  the  result  of  a  TSE)  ,  is  identified  in  need  of  revision  because  of 
changed  conditions  or  requirements,  or  is  considered  to  be  excessively  costly, 
a  Training  Development  Study  (TDS)  may  be  conducted  to  find  the  most  cost- 
effective  way  to  'fix'  the  system.  The  TDS  is  procedurally  similar  to  the 
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Figure  3-7.  TSEA 


CTEA.  Both  require  a  reasonably  accurate  cost  assessment,  the  development  of 
alternative  training  predicted  to  be  effective,  and  a  cost-effectiveness 
analysis.  The  TDS  conducted  for  a  fielded  system  parallels  the  CTEA  conducted 
for  a  developing  system.  Another  specialized  specific  requirement  for  TDS 
exists  in  relation  to  training  devices.  The  two  types  will  be  discussed  in 
the  following  paragraphs. 

3.8.2  TDS,  Fielded  Systems.  The  general  TDS  is  a  study  conducted  to 
develop  a  'fix'  for  the  on-going  training  of  a  fielded  hardware  system  (see 
Figure  3-8).  The  study  is  initiated  when  an  ISTEA,  TSEA,  or  other  training 
subsystem  data  identify  a  need  for  training  revision,  improvement,  or  modi¬ 
fication.  The  TDS  objective  is  to  find  the  most  cost-effective  ways  to 
improve  a  system  found  ineffective,  inefficient,  or  too  expensive.  As  shown 
in  Figure  3-8  ,  any  combination  of  these  conditions  may  be  present  and  confront 
the  TDS  research  team.  The  first  process  is  the  application  of  ISD  ,  Phase  I 
and  Phase  II,  to  the  tasks  or  subsystem  elements  under  study.  When  possible, 
alternative  effective  methods  should  be  developed  since  cost  and  other  con¬ 
straints  (e.g.,  environmental  impact,  trainer  shortages)  may  impact  on  the 
'fix'  selection. 

The  TDS  group  proceeds  to  the  consideration  of  training  devices.  Has  the 
analytic  design  process  suggested  that  a  device  or  simulator  is  a  likely 
effective  training  'fix'?  If  so,  a  training  device  requirement  ^TDR)  ,  a 
formalized  statement  is  drawn  up.  If  approved,  the  TDR  t6  followed  by  a  TDS 
of  the  second  type  (discussed  later  in  Section  3.8.3). 

After  the  proposed  alternative  training  elements  are  designed,  they  are 
the  subject  of  cost-effectiveness  analysis.  In  reality  ,  this  C/E  of  a  training 
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subsystem  is  a  generic  CTEA  and  is  conducted  in  the  same  manner  as  the  pre¬ 
viously  discussed  CTEA  for  a  developing  system  (see  Section  3-3,  Figure  3-2). 
The  results  of  this  CTEA  enable  the  TDS  research  group  to  decide  whether  the 
proposed  fix  (or  fixes)  can  be  expected  to  prwide  the  required  performance 
effectiveness  at  an  affordable  cost.  If  so,  the  alternatives  are  rank-ordered, 
other  issues  addressed,  and  the  best  fix  selected.  If  the  proposed  fixes  do 
not  meet  requirements ,  the  design  process  must  be  repeated. 

At  this  point,  the  TDS  is  completed.  However,  within  the  TRADOC  TEA 
system,  a  follow-on  to  the  TDS  is  implied.  The  selected  training  fix  is 
developed  and  Implemented  according  to  ISD  Phase  II  and  IV  procedures.  It  is 
then  monitored  ,  ISD  Phase  V  ,  to  watch  for  problems  in  Implementation.  The 
TRADOC  TEA  Handbook  (TRASANA,  1979)  recommends  that  an  ISTEA  be  conducted 
after  the  revised  training  has  been  in  operation  for  a  year.  The  ISTEA  will 
determine  if  the  fix  eliminates  or  adds  to  the  performance  gap. 

3.8.3  Training  Device  TDS.  When  a  need  is  formally  recognized  for  a 
training  device  (the  result  of  one  of  several  types  of  initiating  documents)  a 
sequential  series  of  TDS  are  required.  These  TDS  are  input  to  the  requirements 
documents  for  low  risk,  less  expensive  devices  or  input  to  CTEA  and  COEA  for 
hardware  systems.  A  TDS  is  required  before  each  device  development  decision 
point  (ATSC ,  1980;  TRADOC  Cir.  350-4,  1979).  Each  TDS  requires  an  assessment 
of  potential  device  training  effectiveness  as  well  as  a  determination  of 
non-quantif iable  benefits  and  costs.  The  ATSC  Guide  (1980)  suggests  that, 
when  possible,  TDS  predict  proposed  device  effectiveness  by  comparison  to 
methods  currently  in  use  for  training  similar  tasks.  Table  3-3  displays  the 
requirements  for  conducting  device-related  TDS. 
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Table  3-3  Decision  Points  Requiring  TDS ,  Time-Phased 


TDS  TYPE 

LIFE  CYCLE  PHASE 

DECISION/DOCUMENT 
REQUIRING  TDE  INPUT 

Preliminary 

Conceptual 

TDR/ TDLR  (full  LCSMM) 
TDLOA 

Validation 

TDR/ TDLR  (partial  LCSMM) 

Update  of 
Preliminary 
(follows  DT  1/ OT  I) 

Validation 

TDR/ TDLR  (full  LCSMM) 

Final  (follows 

DT  11/ OT  II) 

Full  Scale  Development 

DEVA  IPR 

Abbreviations:  DEVA  IPR  -  development  inprocess  review 
DT  -  development  test 

LCSMM  -  life  cycle  systems  management  model 
OT  -  operational  test 

TDLOA  -  training  device  letter  of  approval 
TDLR  -  training  device  letter  requirement 
TDR  -  training  device  requirement 
TDS  -  training  development  study 


The  device  acquisition  process,  Including  detailed  descriptions  of 
related  study  requirements,  may  be  found  in  such  publications  as  The  Training 
Acquisition  Handbook  (U.S.  Army  CORADCOM,  1978)  and  Training  Device  Require¬ 
ments  Documents  Guide  (PM  TRADE,  1979).  Figure  3-9  shows  a  simplified, 
general  model  of  the  device-related  ITS.  Similar  to  the  CTEA  (Figure  3-2)  ,  it 
identifies  points  where  requisite  processes  are  applied.  The  TDS  begins  at  a 
point  determined  by  the  answers  to  a  sequential  series  of  four  questions. 

Each  negative  answer  sets  the  TDS  initiation  in  an  earlier,  more  speculative 
context.  The  model  shows  how  each  TDS  builds  on  and  updates  earlier  efforts. 

The  first  TDS  concern  is  whether  a  device  already  exists  that  can  be  used 
to  meet  the  TDR.  If  the  device  exists,  the  next  concern  is  to  obtain  empiri- 
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Figure  3-9.  Model  of  a  Training  Device  -  Related  TDS 


cal  data  that  permits  determination  of  effectiveness.  If  these  do  not  exist 
(or  cannot  be  readily  and  economically  obtained)  ,  the  TDS  research  group 
employs  one  of  the  methods  of  effectiveness  prediction.  The  most  likely  are 
device-specific,  such  as  TRAINV1CE  I  (Wheaton  et  al.  ,  1977),  TRAINVICE  II 
(Narva,  1979)*. 

If  a  suitable  device  and  empirical  data  do  not  exist  but  the  device 
concept  is  developed,  then  one  of  the  predictive  methods  may  be  employed  to 
estimate  training  effectiveness  on  the  specified  tasks.  When  the  device  does 
not  exist  and  has  not  yet  been  defined,  the  TDS  research  group  begins  work  by 
looking  at  tasks.  A  well-specified  task  list  permits  the  TDS  to  start  with 
application  of  a  device  prescriptive  method.  At  yet  an  earlier  stage,  the 
operational  system  concept  may  be  developed  but  operator/maintainer  task 
lists  not  yet  compiled.  Here  the  TDS  process  begins  with  a  task/cue  and 
response  analysis. 

In  what  might  be  considered  "worst  case",  at  the  earliest  development, 
the  TDS  research  group  may  be  dealing  with  an  operational  system  still  in  the 
concept  development  stage.  Sometimes  a  need  for  concurrent  device  development 
is  apparent  at  this  early  LCSMM  phase  (e.g.  ,  aircraft,  tanks,  air  defense 
weapons)  because  of  known  exorbitant  operating  or  ammunition  costs  or  high 
risk  to  operators.  The  team  commences  by  conducting  a  front-end  analysis 
(FEA)  to  determine  the  tasks  to  be  trained  on  the  device. 

Depending  upon  the  starting  point,  the  sequence  of  analysis  is  (1) 
determination  of  tasks,  (2)  task/cue  and  response  analysis,  (3)  prescription 
of  effective  devices,  and  (4)  prediction  of  device  effectiveness.  When  this 


*For  a  description,  see  Section  2.4.2,  Predictive  Methods. 
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point  Is  reached,  the  TDS  team  compares  and  ranks  the  alternatives,  taking 
Into  consideration  training  effectiveness,  cost,  and  any  other  non-quantifi- 
able  benefits. 

3.9  Summary 

This  discussion  has  described  a  systematic  approach  to  the  selection  and 
application  of  CTEA  models  appropriate  to  the  analytic  requirement  related  to 
a  variety  of  training  systems  analysis  needs.  The  general  model  of  training 
system  analysis  (Figure  3-1,  above),  depicts  the  systematic  logical  analysis 
that  directs  the  analyst  to  the  specific  CTEA  submodel  most  appropriate  to  the 
task  at  hand.  This  logical  analysis  and  its  related  analytic  submodels  have 
been  described  in  some  detail  in  the  foregoing  sections.  This  systematic 
approach  applies  to  the  selection  and  conduct  of  CTEAs  for  various  training 
systems  development  and  evaluation  problems.  This  approach  can  be  applied  by 
analysts  at  various  levels  and  agencies  throughout  the  Army  training  and 
combat  development  structure.  If  it  is  systematically  applied,  the  approach 
should  lead  directly  and  efficiently  to  the  most  appropriate  analytic  design 
for  any  given  CTEA  problem  under  diverse  system  situations. 

While  no  truly  generic  model  for  CTEA  has  been  identified  or  developed  in 
this  research,  the  systematic  classification  of  the  various  submodels  in 
relation  to  the  problems  to  be  handled  may  be  considered  a  substitute  for  such 
a  generic  model.  This  systematic  approach  amounts  to  the  currently  achievable 
state  of  the  art  in  the  area  of  general  CTEA  methodology  at  this  time.  It  is 
believed  that  further  developments  will  require  breakthroughs  in  analytic 
thought  and  in  methods  generalization  that  cannot  now  be  defined.  Even  so, 
the  approach  outlined  abcwe  can  be  a  useful  tool  to  training  and  system 
developers  if  they  will  apply  it  to  their  specific  analytic  problems. 
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APPENDIX  A 


ABSTRACTS 


Jacobs,  T.O.  and  Hardy,  R,  A,,  Jr,  Test  and  evaluation  of  training  extension 
course  CTEC),  HjjidRRO  TR-74-16,  Alexandria,  YA:  Human  Resources  Organization, 
June  1974. 


This  research  was  a  pilot  study  designed  to  provide  an  inital  evaluation 
of  the  effectiveness  of  TEC  as  training  medium.  At  the  time  the  TEC  concept 
was  being  implemented,  planning  was  also  under  way  for  testing  the  1972  MOS 
11B40  Proficiency  Pay,  TEC  training  was  provided  to  soldiers  taking  HOS 
11B40  tests  and  the  MOS  test  scores  were  used  as  a  criterion  of  effectiveness. 
Fifty-six  TEC  lessons  covering  four  major  MOS  test  areas  were  prepared  by 
the  U.S.  Array  Infantry  School,  Control  battalions  were  given  all  preparatory 
material  and  media  except  TEC  while  "TEC"  battalions  received  all  these  plus 
TEC  material,  A  second  control  group  was  drawn  from  Army-wide  test  takers 
to  ensure  that  the  battalion  control  groups  represented  the  Army  as  a  whole. 
An  additional  correlation  of  MOS  test  score  to  verbal  English  (VE)  score 
was  also  made, 

Major  Findings: 

(1)  Correlation  of  MOS  test  score  with  VE  scores  was  low  -  .3) 
but  significant.  Therefore,  it  was  used  as  a  control  during 
analysis  of  results, 

(2)  Analysis  of  relationship  of  TEC  and  verbal  ability  suggests 
TEC  did  not  completely  reach  the  target  population.  Low  users 
of  TEC  were  in  unit  with  low  average  verbal  ability;  high  users, 
high  verbal  ability. 

(3)  Users  of  TEC  scored  significantly  better  in  Area  One  -  Individual 
Weapons  and  displayed  a  trend  (pot  consistent)  for  better  scores 
in  Area  Three  -  Tactics. 

(4)  An  analysis  of  lessons  studied  did  not  show  that  TEC  battalion 
members  had  a  greater  chance  of  answering  correctly  questions 
covered  by  TEC  material  than  questions  not  so  covered. 

(5)  Amount  of  studying  was  positively  and  significantly  correlated 
with  MOS  scores  (_»3).  It  remained  significant  when  VE  was  con¬ 
trolled  (.2),  Tills  conflicts  with  the  preceding  finding  and 
suggests  that  there  may  have  been  differences  between  battalions 
in  the  manner  of  using  materials  and  that  these  differences  were 
strong  enough  to  influence  the  outcome, 

(6)  The  rank  order  correlation  between  the  average  number  of  lessons 
studied  by  battalion  and  the  average  MOS  test  score  by  battalion 
was  .88  -  a  significant  figure. 

(7)  Reasons  suggested  for  non-use  of  TEC  were:  lack  of  emphasis 
within  the  unit,  competing  programs  and  priorities,  and  lack 

of  time  during  the  duty  day.  All  were  deemed  sufficient  reason 
for  decreasing  TEC  usage. 


The  research  conclusions  were; 


(1)  MOS  test  performance  was  higher  in  units  in  which  TEC  materials 
were  available  and  which  at  the  same  time  reported  high  use  of 
TEC  materials,  This  is  held  as  "suggesting"  that  TEC  materials 
were  useful  in  preparing  for  the  MOS  test, 

(2)  Analysis  of  both  the  MOS  test  and  the  TEC  materials  suggest 
that  a  penalty  was  paid  for  the  urgent  time  pressures  imposed 
during  the  initial  implementation  of  the  TEC  concept.  Further 
recommendations  axe  made  for  improving  both  the  material  de¬ 
velopment  and  pretesting  procedures. 


Table  1 


Pay  Grade  and  Unit  Distribution  of  Subjects 
Participating  in  the  TEC  Evaluation 


Group 

Unit 

Battalion  j 

Pay  Grade 

Total 

E-5 

E-6 

E-7  | 

TEC 

82  d 

2d  BN,  325th.  Infantry 

30 

29 

2 

61 

experimental 

2d  3N,  506tlL  Infantry 

54 

23 

5 

82 

(Sound/Slide) 

2d  BN,  504th.  Infantry 

30 

21 

0 

51 

All 

114 

73 

7 

194 

4  th. 

2d  BN,  8th.  Infantry 

16 

5 

0 

21 

1st  BN,  11th  Infantry 

5 

6 

1 

12 

All 

21 

11 

1 

33 

2d 

1st  BN,  41st  Infantry 

13 

19 

0 

37 

2d  BN,  50th  Infantry 

9 

7 

2 

13 

All 

27 

26 

2 

55 

30th  (NG) 

1st  BN,  119th  Infantry 

31 

5 

0. 

36 

7 1st  QvG) 

2d  BN,  143d  Infantry 

14 

3 

3 

20 

73rd (NG) 

1st  BN,  166th.  Infantry 

27 

9 

5 

41 

Control 

82  d 

2d  BN,  505th.  Infantry 

3 

5 

3 

11 

(No  sound/ 

1st  BN,  508th  Infantry 

15 

19 

6 

40 

Side) 

All 

18 

24 

9 

51 

4  th 

1st  BN,  10th.  Infantry 

9 

13 

0 

22 

2d 

2d  BN,  41st  Infantry 

7 

4 

1 

12 

1st  BN,  50th  Infantry 

7 

12 

3 

22 

All 

14 

16 

4 

34 

30th  G'lG) 

1st  BN,  118th  Infantry 

24 

5 

0 

29 

1st  BN,  120th  Infantry 

22 

7 

2 

31 

All 

46 

12 

2 

60. 

71st(NG) 

1st  BN,  143rd  Infantry 

16 

5 

0 

21 

3d  BN,  143rd  Infantry 

13 

5 

4 

22 

All 

29 

10 

4 

43 

TEC  users:  n  =  379 
Control:  n  =  183 


n  for  the  Amy-wide  sample  is  not  stafed. 


Indermill,  K.  ,  Gale,  P.,  and  Reeder,  J.  How  to  design  and  develop  self  and 
supervised  Instruction;  A  gnlde  for  developing  correspondence  Instruction. 
Tallahassee,  7L:  The  Center  for  Educational  Technology,  Florida  State 
University,  1  February  1975. 

Under  the  sponsorship  of  the  TJ.S.  Army  Combat  Arras  Training  Board  at 
Fort  Benning,  GA,  Indermill  and  her  associates  have  developed  a  proceduralized 
guide  for  developing  correspondence  course  lessons  with  an  on-the-job  training 
( OJT)  option.  The  manual  covers:  0-)  how  to  write  OJT  lessons;  (.2)  how  to 
write  training  objectives  in  standard  format;  (3)  how  to  write  interactive 
instruction;  and,  (4)  how  to  validate  newly-prepared  instructional  material. 
Each  of  these  sections  provides  the  lesson  developer  with  instructions,  practice 
lessons,  self-evaluation  tests,  checklists,  and  procedural  guides. 

The  authors  recognize  the  similarity  to  TEC  lessons.  They  state: 

Many  of  the  design  techniques,  currently  used  in  the  Training 
Extension  Course  (TEC)  self-study  program,  such  as  behavioral 
objectives,  interactive  design,  and  self-evaluation  progress 
checks,  need  to  be  brought  to  bear  on  our  correspondence 
instruction. 

This  method  is  efficient  since  it  takes  advantage  of  expensive  prior  TEC 
lesson  development.  They  further  state  that,  as  a  minimum,  each  TEC  lesson 
should  be  converted  to  a  correspondence  lesson  thereby  reaching  a  larger, 
non-hardware  dependent  audience.  To  this  end,  a  chapter  is  included  that 
teaches  the  training  developer  how  to  convert  TEC  material  to  a  correspondence 
lesson. 

Two  sample  lessons  complete  the  guide.  One  is  a  revised  TEC  lesson  with 
an  OJT  option  —  Sign-?!  Subcourse  000  (OJT  only),  Radio  Set  AN/PRC-77.  The 
other,  Infantry  Subcourse  3-2,  Map  Reading  was  not  prepared  from  a  TEC  lesson 
and  does  not  offer  an  OJT  option.  The  material  in  the  former  lesson  is  far 
superior  to  the  latter.  Both  were  prepared  by  participants  in  the  develop¬ 
mental  testing  of  the  manual.  It  is  not  stated  whether  or  not  the  lessons 
were  validated. 


McCluskey,  M.  R.  and  Tripp,  J.  M.  An  evaluation  of  the  utilization,  maintenance, 
and  perceived  benefits  of  the  training  extension  course  (TEC).  HuroRRO  TR-75-18. 
Alexandria,  VA:  Human  Resource  Organization,  June  1975. 


The  purpose  of  this  research  was  to  evaluate  the  program  designed  to  imple¬ 
ment  TEC.  Data  were  gathered  tc  provide  the  CATR  with  information  to  identify 
possible  alternative  actions  for  improving  the  implementation,  management,  and 
distribution  of  TEC  on  an  Army-wide  basis.  The  evaluation  consisted  of  gathering 
data  on  the  utilization,  maintenance,  and  perceived  benefits  of  the  TEC  system. 
These  data  were  collected  by  means  of  questionnaires,  interviews,  maintenance 
forms,  and  a  utilization  form.  A  survey  of  the  initial  group  of  Active  Army 
and  Reserve  Component  units  that  received  TEC  showed  that  desired  levels  of 
usage  had  not  been  reached  because  of  a  lack  of  command  emphasis. 

Conclusions  were: 

(1)  TEC  had  not  been  implemented  for  a  sufficient  length  of  time 
to  permit  an  adequate  evaluation. 

(2)  An  adequate  library  of  lessons  was  not  available  for  meaningful 
study  in  terms  of  either  MOS  proficiency  or  accomplishment  of 
unit  training  goals. 

(3)  Pesults  were  biased  since  one  unit  accounted  for  80%  of  the 
TEC  usage. 

(4)  During  the  period  of  observation  there  was  an  increased  usage 
of  TEC  in  the  group  mode, 

(5)  Command  emphasis  and  information  concerning  the  TEC  system  were 
reduced  in  content  and  importance  during  transmission  down 
through  the  chain  of  command. 

(6)  Attitudes  of  trainers  and  users  were  moderately  positive. 

(7)  Battalion  level  TEC  learning  centers  did  not  appear  to  be  the 
most  appropriate  or  effective  level  for  distribution  of  TEC 
materials. 

(3)  Personal  interview  suggested  that  TEC  utilization  might  be  in¬ 
creased  by  thorough  promotion  at  the  company  level  and  by  dividing 
the  TEC  hardware  and  software  allocation  between  battalion  and 
company  levels.  If  equipment  is  to  be  used  for  groups,  the  pro¬ 
jection  capability  should  be  increased  to  30-200  person  suitability. 
Learning  centers  should  have  full-time  personnel  assigned  and  an 
operating  budget.  A  system  of  rewards  and  incentives  for  trainers 
and  users  might  increase  usage;  and  the  final  suggestion,  some  of 
the  simple  maintenance  should  be  decentralized  to  the  battalion 
level. 
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Department  of  Army,  "TEC  cost  and  qperational  eff ectiyeness  analysis  (TEC  COEA) 
in  TRADOC  TRADER  format."  Fort  Henning,  GA:  Corohat  Arms  Training  Board. 

4  August  1975. 

When  the  TEC  system  was  in  its  acquistion  stage,  a  cost  and  operational 
effectiveness  analysis  (COEA)  for  TEC  was  conducted  for  TRADOC  TRADER  by  the 
Combat  Arms  Training  Board  CCATB),  Fort  Benning,  Georgia.  The  baseline  was 
the  then-current  training  system  in  individual  skills  at  the  unit  level, 
described  as  carried  out  By  unit  officers  and  NCOs.  It  was  further  described 
as  follows: 

Usually,  the  designated  unit  instructor  develops  a  custom- 
made  program  of  instruction  (POIl  using  available  training 
literature  (which  is  often  outdated),  and  whatever  training 
aids  are  available.  The  class  is  normally  presented  by 
lecturer  and  the  POT  is  seldom  used  again.  Little  diagnos¬ 
tic  testing  or  post-instruction  performance  testing  is  ever 
planned  or  conducted. 

The  proposed  alternative  was  the  TEC  Training  System(here  training  system  is 
used  in  a  non-specific  or  generic  sense).  It  was  described  as  multimedia 
standardized  instructional  materials  prepared  by  service  schools  for  use  in 
the  field.  The  TEC  material  was  viewed  as  designed  for  either  small  group  or 
individual  self-paced  mode. 

No  new  research  was  conducted  by  the  COEA  study  group.  They  concluded 
that  TEC  was  an  extremely  effective  system  based  on  an  ARI  COEA  conducted 
for  DCSPER.  The  effectiveness  part  of  this  study  was  the  work  conducted  by 
Knerr  et  al.  (1975).  The  ARI  COEA  also  concluded  that  TEC  trained  soldiers, 
both  Active  Array  and  National  Guard,  performed  significantly  better  than  those 
trained  on  conventional  methods  and  at  lesser  cost. 

The  CATB  COEA  conducted  an  indpendent  cost  analysis.  They  compared  costs 
based  on  1064  battalions  using  100  hours  of  annual  instruction.  This  level  of 
implementation  was  predicted  for  19.77.  Start-up  and  initial  distribution  costs 
were  calculated  as  of  1974,  They  also  assumed  an  economic  life  of  C.  1  year  for 
the  baseline  system  since  their  field  research  showed  that  unit  instruction  was 
tailormade  for  each  instructional  situation  and  seldom  recycled.  The  economic 
life  of  TEC  hardware  was  estimated  at  6  years;  software  development  and  revision, 
continuous.  Total  discounted  costs  (FY74-FY80)  were  shown  to  be  almost  lh 
times  higher  for  conventional  training  then  the  proposed  TEC-augmented  training. 
(See  attached  tables.)  The  savings  was  based  on  cost  avoidance  — TEC  was  fore¬ 
seen  as  requiring  less  officer/NCO  support  at  the  battalion  level. 

Non-quantif iahle  elements  were  also  examined  for  the  COEA.  A  consultant's 
report  mentions  emphasis  on  trainee's  needs  rather  than  instructor's!  creation 
of  a  training  R&D  capability  applicable  to  other  future  needs,  and  many  benefi¬ 
cial  effects  at  the  service  schools  and  in  Army  training  in  general.  He  says, 
"Consequently,  the  TEC  system  has  become  a  model  of  (sic)  other  training  oriented 
R&D  projects  within  TRADOC." 


The  COEA  study  groups  conclusions  were  as  follows; 


1.  RELATIVE  EFFECTIVENESS 

OVERALL  EFFECTIVENESS  vCOST  ($  MILLION) 

TEC  87.5*  42.7 

CONVENTIONAL  67.5*  50.5 

*Active  Army  and  National  Guard  averaged  (Equal  N’s  assumed). 


Relative  effectiveness  (TEC  vs  CONVENTIONAL)  - 
87.5/67.5  =  1.30 

2 .  RELATIVE  COST 

TEC/ CONVENTIONAL  =  42,7/50.5  =  .84 

3.  RELATIVE  WORTH 

TEC/ CONVENTIONAL  -  1.30/. 34  ■  1.54 

4.  CONCLUSIONS  OF  THE  ART  STUDY 

1.  On  the  average  TEC  instruction  teaches  both  written  posttests  and 
hands  on  performance  tests  better  than  conventional,  live  instructor  instruction. 

2.  The  state  of  training  in  individual  skills  in  both  the  Active  Army 
and  National  Guard  is  generally  poor. 

3.  TEC  teaches  soldiers  with  low  abilities  and  high  abilities  equally 
well,  whereas  conventional  instruction  is  not  as  effective  with  low  ability 
soldiers  as  it  is  with  high  ability  soldiers.  Over  all  ability  groups,  TEC 
teaches  "better." 

4.  Considering  all  costs,  TEC  is  more  cost-effective  than  conventional 
instruction  in  units. 
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Temkins,  S.,  Connally,  J.  A.,  Marvin,  M.D.,  Valdes,  A.  L.,  and  Caviness,  J.  A. 
A  cost  assessment  of  Army  training  alternatives.  Research  Problem  Review  75-3. 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences,  August  1975. 


This  cost  assessment  was  undertaken  specifically  to  provide  comparative 
cost  data  on  TEC  for  individualized  instruction  and  on  equivalent  conventional 
classroom  Army  group  instruction  as  a  part  of  a  cost-effectiveness  study  of 
TEC.  Costs  were  estimated  based  on  historical  data  and  tabulated  for  develop¬ 
mental  and  operational  use  of  typical  TEC  lessons  and  conventional  group  in¬ 
struction.  The  two  were  compared  in  a  "comparison  module"  (defined  as  100 
lessons  per  year  to  each  E3-E4  in  a  standard  infantry  battalion)  . 

The  findings  were  as  follows: 

The  development  cost  for  TEC  was  estimated  at  $15,920  per  lesson.  Opera¬ 
tional  cost  was  estimated  at  $38,279  per  battalion  (533  men).  The  conventional 
system  has  no  development  costs;  operational  costs  equivalent  to  100  TEC  lessons 
were  estimated  at  $47,437  per  battalion,  chiefly  instructors'  time.  Under  a 
projected  expanded  utilization  to  1,064  battalions  per  year,  TEC  annual  costs 
(including  lesson  development  and  revision)  -,/ould  total  $42,100,000  against 
projected  conventional  annual  cost  of  $50,473,000.  Based  on  an  estimated  5-  * 
year  lesson  life,  development  costs  under  expanded  utilization  are  estimated 
at  about  $600  per  battalion  to  200  lessons,  with  an  additional  estimated  annual 
cost  for  revisions  of  $690  (1975  dollars)  per  battalion. 

The  authors  concluded  that  use  of  TEC  individualized  instruction  with  either 
cassette  tape  or  audiovisual  aids  is  projected  to  be  less  costly  in  comparison 
to  conventional  instruction  as  volume  of  use  increases. 

The  finding  assumes  optimal  use  of  TEC  by  individuals.  Group  use  although 
seen  as  a  possibility  is  not  considered  nor  is  under-utilization  or  cost  effec¬ 
tiveness.  However,  the  caveat  was  expressed  that  the  level  of  implementation 
must  be  as  assumed  in  the  study  to  effect  an  annual  cost  savings. 

The  developmental  costs  are  identified  by  the  following  categories: 

Job  Analysis 

1.  Selection  of  Duty  Position 

2.  Job  ID 

3.  Preparation  of  Task  Statements 

4.  Preparation  of  Task  Inventory  (lists) 

5.  Selection  of  Ta.ks  for  Training  Development 

6.  Job  Task  Data  Cards 

7.  Selection  of  Common  Tasks 

8.  Field  Validation 

9 .  Rev  iews 


A- 9 


Curriculum  Design 


1.  Coordinate  Summary  Test 

2.  Prepare.  Lesson  Tests  and  Treatment 

3.  Draft  Scripts 

4.  Prepare  Storyhoards 

5.  Motion  Pictures 

6.  Group  Trails 

7.  Art  Work 

3.  Retrial  Cycle 


Curriculum  Review 


1.  CSS  Review  Board 

2.  Final  CSS  and  GFM  Review  Board 

3.  Review  and  Analysis 

4.  Lesson  Treatment 

5.  Script 

6.  Storyboard 

7.  Post  Individual  Trial  Analysis 

8.  Final  Storyboard  Review 

9.  Group  Trial  Analysis 

10.  Final  Audio  Review 

11.  Final  Art  Review 

12.  Answer  Print  Review  and  Approval 
(35ram  and  Super  8nm) 


Operational  costs  were  accumulated  in  these  categories: 

(1)  At  combat  arms  school 
-job  analysis 

-curriculum  design  and  follow-through. 

-curriculum  reviews 

(2)  At  USACATB 

-hardware  and  equipment  for  TEC  system 
-development 

(3)  Lesson  contractors 

(4)  Lesson  masters 

(5)  Reviews 

Conventional  group  training  was  the  only  instructional  alternative  compared 
to  TEC.  No  developmental  costs  were  included  (sunk  costs).  Operational  cost 
categories  were  as  shown  in  Table  1.  Cost  for  all  activities  common  to  both 
groups  were  excluded. 
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TABLE  1  COSTS  OF  PREPARATION  TIME,  CLASS  THE,  AND  INSTRUCTIONAL  SUPPORT  TIME 
)R  10  CCNVENTIONA!  GROUP  INSTRUCTION  CASES  FOR  CONTENT  OCT  PARABLE  TO  TEC  LESSONS:  IN  1975  DOLLARS 


COMPARABLE  TEC 
LESSON  SUPER 
BASE 

NUMBER  OF 
TRAINEES 

INSTRUCTOR  CLASS 
PREPARATION  THE  COST 
(1975  DOLLARS) 

INSTRUCTOR  CLASS 
THE  OOST 
(1975  DOLLARS) 

L'PPORT  TILE  FOR 
INSTRUCTOR  OOS1 
(IN  1975  DOLLARS) 

•TOTAL 

(IN  1975  DOLLARS) 

4 

17 

9.47 

34.75 

21.90 

66.12 

6 

10 

10.68 

25.67 

15.13 

51.48 

S 

8 

35.00 

31.50 

62.78 

129.28 

5 

10 

22.15 

53.16 

22.80 

98.11 

2 

10 

12.62 

15.78 

19.89 

48.29 

2 

20 

22.59 

10.04 

37.77 

70.40 

3 

20 

22.60 

10.17 

12.40 

45.17 

3 

10 

21.00 

14.00 

10.67 

45.67 

4 

7 

20.04 

S.01 

30.04 

55.09 

4 

15 

28.25 

10.17 

30.06 

68.48 

Lesson  per  soldier  cost 

5  0.41 

$  0.42 

$  0.53 

$  1.48 

Source:  ARI/HRU,  Fort  Benning 
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Some  other  concerns  were  addressed.  One  was  the  implication  of  doctrinal 
change.  The  possibility  was  anticipated  at  different  rates  by  four  schools 
queried  and  a  factor  to  estimate  TEC  revision  costs  included  in  calculations. 
Inflation  factors  were  applied. 

Another  concern  was  the  effect  of  technological  ohsolescence.  USCATB 
foresaw  that  the  Besler  CUE/SEE  and  tape  cassette  player  would  he  adequate 
for  "another  ten  years"  (i.e.,  1984-85i.  They  felt  that  "only  a  progression 
to  other  instructional  techniques,  e.g.,  interactive  branching,  would  require 
a  different  kind  of  equipment". 
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Goetz,  J.A.,  LTC,  "TEOyalidated  service  school  instruction  at  the  unit  level". 
A  paper  presented  at  the  8th  NTEC/Tndustry  Conference  and  published  in  Pro¬ 
ceedings:  New  concepts  for  training  systems..  Orlando,  FL;  Naval  Training 
Equipment  Center,  18-20  November  1275. 

In  this  symposium  presentation  LTC  Goetz  presents  a  history  and  excellent 
description  of  the  TEC  program  (to  date  1975).  He  reports  the  results  of  TEC 
validation.  Forty-eight  Armor,  27  Infantry,  .53  Field  Artillery,  and  53  Air 
Defense  Artillery  lessons  were  evaluated  by  students.  Using  the  TEC  Post 
test,  students  averaged  ^9.1%  Items  passed  before  training  and  91.9%  passed 
after  training.  He  concludes  that  the  "validation  procedure  in  effect 
guarantees  that  TEC  lesson  (sic)  will  teach." 
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Knerr,  C.S.,  Downey,  R.G.,  and  Kessler ,  J.J.  Training  Individuals  in  Army 
units:  TEC  iessons-  and  conventional  methods.  Research  Report  1188. 
Alexandria,  VA:  U.S.  Amy  Research  Institute  for  the  Behavioral  and  Social 
Sciences.  December  1975. 


The  Army  has  developed  a  system  of  self-paced  audiovisual  lessons — the 
Training  Extension  Course  (TEC)  —  designed  to  upgrade  individual  skills  and 
to  help  commanders  conduct  individual  proficiency  training.  This  report 
compares  the  effectiveness  of  TEC  training  with  that  of  conventional .  Array 
classroom  instruction,  evaluating  both  against  a  baseline  level  of  performance. 

Five  TEC  lesson  categories  (courses)  were  used,  four  from  infantry 
training  and  one  from  field  artillery  training.  Five  hundred  eligible  soldiers 
were  selected  by  their  units  for  training  at  two  Amy  posts  and  from  a  National 
Guard  division.  Each  soldier  was  assigned  to  one  course.  Participants  were 
randomly  divided  into  three  equal  groups.  The  first  group  received  the  TEC 
lessons;  the  second  group  received  conventional  instruction  (Cl)  on  the  same 
topics;  and  the  third,  baseline  (BL),  group  received  no  special  training 
during  the  research. 

Following  administration  of  a  pretest,  a  hand-on  performance  test  for 
each  course  was  given  all  participants  after  the  training  to  measure  the 
comparative  effectiveness  of  each  training  method. 

The  TEC-trained  groups  in  general  scored  highest  in  the  performance  tests, 
followed  by  the  Cl  groups;  KL  groups  performed  least  well.  The  TEC  groups 
performed  consistently  better  than  the  BL  group.  The  TEC  groups  performed 
better  than  either  the  Cl  or  BL  groups  on  tests  that  emphasized  reasoning  and 
information.  TEC  and  Cl  groups  performed  equally  well,  and  better  than  the 
BL  groups,  on  performance  tests  that  emphasized  skill  with  equipment  and 
psychomotor  activities. 

In  the  Active  Army,  performance  test  scores  in  the  TEC  groups  were  equally 
high  for  persons  with  low  and  high  general  mental  ability,  as  measured  by  the 
GT  aptitude  area  score.  In  contrast,  performance  test  scores  for  Cl  groups 
were  generally  lower  for  persons  with  lower  general  mental  ahility. 

TEC  lessons  consistently  improved  soldier  performance  regardless  of  the 
soldiers'  level  of  mental  ability,  suggesting  TEC  would  be  particularly  useful 
for  training  of  mixed-aptitude  personnel.  Where  skill  or  practice  with  equip¬ 
ment  is  involved,  however,  conventional  instruction  i$  equally  effective. 
Results  are  expected  to  generalize  to  other  situations,  as  findings  were 
basically  the  same  in  the  three  different  sites  of  this  research. 

The  number  in  each  experimental  group  and  other  relevant  statistics  are 
as  shewn  in  the  following  tables. 
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(Knerr)  Table  C-10 


ACTIVE  ARMY  CORRELATIONS  AMONG  PERFORMANCE  TEST, 
LAI  PRETEST,  AND  GT  SCORES 


Lesson 

Training 

Performance 

Performance 

LAI  Pretest 

Category 

Group 

and  LAI  Pretest 

and  GT 

and  GT 

N 

Hand 

TEC 

.62* 

-.02 

.40 

13 

Grenades 

Cl 

.83* 

.61* 

.69* 

15 

BL 

.41 

.24 

.60* 

20 

Combined: 

.47* 

.16 

.52* 

48 

LAW 

TEC 

.32 

.06 

.39 

18 

Cl 

.30 

.54* 

.36 

16 

BL 

.57* 

.44* 

.72* 

19 

Combined: 

.42 

.30* 

.44* 

53 

M16A1 

TEC 

.15 

.02 

.48* 

26 

Rifle 

Cl 

.51* 

.24 

.46* 

22 

BL 

.37 

.43* 

.21 

26 

Combined: 

.36* 

.21 

.36* 

74 

Mortar 

TEC 

.76* 

-.13 

.26 

17 

FDC 

Cl 

.63* 

.27 

.21 

18 

BL 

.48 

.71* 

.66* 

16 

Combined: 

.66* 

.26 

.44* 

51 

Surveyed 

TEC 

.51* 

.02 

.13 

21 

Firing 

Cl 

.77* 

.11 

.35 

20 

Charts 

BL 

.44 

.27 

.48* 

18 

Comb ined: 

.56* 

.09 

.32* 

59 

*p<.  05 
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(Knerr)  Table  C-ll 


ACTIVE  ARMY  PERCENT  CORRECT  ON 
LAI  PRETESTS  AND  POSTTESTS 


Lesson 

Training 

Percent  Correct 

Category 

Group 

LAI  Pretest  LAI  Posttest 

Hand 

i’EC 

46% 

88% 

Grenades 

Cl 

42% 

54% 

LAW 

TEC 

43% 

83% 

Cl 

44% 

56% 

M16A1 

TEC 

44% 

78% 

Rifle 

Cl 

38% 

49% 

Mortar 

TEC 

73% 

94% 

FDD* 

Cl 

57% 

72% 

Surveyed 

TEC 

42% 

67% 

Firing 

Charts 

Cl 

43% 

57% 

*Lesson  No.  010-071-6601  tests  only 


Knerr  (Table  D-9) 


NATIONAL  GUARD  CORRELATIONS  AMONG  PERFORMANCE  TEST, 
LAI  PRETEST,  AND  GT  SCORES 


Lesson 

Training 

Performance 

Performance 

LAI  Pretest 

Category 

Group 

and  LAI  Pretest 

and  GT 

and  GT 

N 

Hand 

TEC 

.05 

.58* 

.40 

20 

Grenades 

CT 

.73* 

.31 

.21 

20 

BL 

.58* 

.36 

.63* 

19 

Combined: 

.46* 

.32* 

.40* 

59 

LAW 

TEC 

.25 

.13 

.06 

19 

Cl 

.27 

.15 

.44 

20 

BL 

.08 

.18 

.03 

20 

Combined: 

.09 

.18 

.20 

59 

M16A1 

TEC 

.14 

• 

ro 

.23 

15 

Rifle 

Cl 

.45 

.37 

.36 

15 

BL 

.04 

-.01 

.46 

13 

Combined: 

.36* 

• 

o 

00 

.21 

43 

Mortar 

TEC 

.48* 

.26 

.31 

18 

FDC 

Cl 

.32 

.49* 

.43 

17 

BL 

.42 

.31 

.41 

19 

Combined 


41* 


32* 


36* 


54 


(Knerr)  Table  D-10 


NATIONAL  GUARD  PERCENT 


:t  ON  LAI 


qnP  posttests 


Lesson 

Training 

Percent 

Correct 

Category 

Group 

LAI  Pretest 

LAI  Posttest 

Hand 

TEC 

39% 

89% 

Grenades 

Cl 

36% 

37% 

LAW 

TEC 

40% 

83% 

Cl 

42% 

51% 

M16A1 

TEC 

48% 

82% 

Rifle 

Cl 

44% 

57% 

Mortar 

TEC 

52% 

83% 

FDC  *■ 

Cl 

42% 

69% 

*Lesson  No.  010-071-6601  tests  only. 
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Leonard,  R.L.,  Jr.,  Wheaton,  G.R.,  and  Cohen,  F.P.  Transfer  of  training  and 
skill  retention.  TR-76-A3.  Alexandria,  VA:  U.S.  Army  Institute  for  tne 
Behavioral  and  Social  Sciences,  October  1976. 


The  objective  of  this  research  was  the  exploration  of  the  retention  of 
training  content  and  transfer  of  that  training  to  a  criterion  performance  test. 
The  research  team  also  explored  the  impact  of  refresher  training  on  transfer. 

A  three-phase  experiment  was  carried  out  over  a  17-week  period  using  106  Army 
personnel  divided  into  six  groups.  The  original  training  was  provided  by  TEC 
lessons  dealing  with  selection,  maintenance,  and  use  of  hand  grenades.  Three 
groups  were  given  refresher  training  by  re-exposure  to  the  TEC  materials  at 
varying  Intervals  from  the  original  training. 

Findings  provided  empirical  support  for  the  Army's  policy  of  evaluating 
training  programs  with  performance  oriented  criterion  tests. 

Of  relevance  to  a  study  of  TEC,  the  research  noted  several  problems. 

Quoting  from  their  summary: 

Finally,  several  problems  that  were  encountered  with  the 
TEC  training  materials  are  worth  mentioning.  The  Bessler 
Cue-See  machines  employ  separate  tapes  for  the  audio  and 
visual  portions  of  the  program.  The  video  portion  is  re¬ 
wound  completely  by  pushing  a  single  button  on  the  front 
of  the  machine  while  the  audio  portion  can  be  rewound  par- 
tailly  just  like  an  ordinary  tape  recorder.  Unfortunately, 
the  video  rewind  button  is  located  adjacent  to  the  "continue" 
button  which  the  subject  presses  to  restart  the  program  after 
he  has  answered  a  question.  It  is  quite  easy  to  confuse  the 
buttons  and  once  the  video  rewind  button  has  been  pressed 
there  is  no  alternative  but  to  rewind  the  audio  portion  as 
well  and  start  the  program  over.  Since  the  programs  are  30- 
45  minutes  in  length,  the  necessity  to  start  over  can  waste 
considerable  time.  Also,  the  use  of  separate  video  and  audio 
tapes  makes  it  quite  easy  fcr  the  two  to  be  out  of  phase, 
especially  if  the  audio  tape  is  not  fully  rewound  from  the 
previous  use.  It  would  seem  that  these  features  could  be 
corrected  by  a  fairly  simple  modification  of  the  Bessler  Cue- 
See  machine  since  the  overall  value  of  the  machines  appears 
to  be  high. 

Thib  study  had  some  potential  for  evaluating  the  effectiveness  of  TEC  as 
training  medium.  However,  all  the  subjects  had  previously  received  training  in 
hand  grenades  during  Basic  Combat  Training  and  Advanced  Individual  Training 
using  other  media/methods. 

One  of  the  six  groups  was  designated  an  "untrained"  control.  They  were 
given  a  criterion  referenced  performance  test  (CRT)  followed  immediately  by  TEC 
training  and  a  post-TEC  training  test.  This  group  was  the  only  one  to  be  tested 
before  TEC  training.  Alter  17  weeks  the  control  group  received  TEC  training  as 
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a  refresher,  followed  by  both  tests.  This  control  group  shoved  significantly 
lover  scores  when  results  were  examined  for  transfer  from  TEC  to  four  CRT  sub¬ 
tests  as  a  function  of  time  since  training  and  as  a  function  of  time  between 
Initial  and  refresher.  On  one  category  —  Identifying  components  of  a  hand 
grenade  —  the  difference  was  not  significant.  This  was  blamed  on  poor  coverage 
of  test  Items  In  the  TEC  material.  This  Indicates  that  TEC  is  a  successful 
refresher  training  media,  although  the  final  sample  was  6mall. 
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Swezey,  R.  W.  and  Eakins,  R.  C.  Task  Report  Training  Extension  Courses  Research: 
Task  B-l,  Measures  of  Effectiveness:  Development  of  Performance  Tests  for  Use 
In  Evaluating  the  Training  Extension  Course  Program.  Ft.  Benning,  GA.  Litton 
Mellonics  Systems  Development  Division,  September  1977.  Unpublished  paper. 


This  task  report  is  one  of  a  number  on  the  research  support  provided  to 
the  ARI,  Ft.  Benning  Field  Office.  It  documents  the  development  and  validation 
of  tests  designed  to  measure  the  individual  performances  of  Army  personnel. 

The  authors  describe  the  construction  of  the  tests,  content  validation,  trial 
administration  of  tents,  revisions  made  during  the  tryout,  and  final  revisions. 
Samples  of  the  lessons  and  the  performance  measures  are  shown  in  Table  1  and  2. 

Five  independent  performance  tests  were  developed  for  use  as  criterion 
variables  in  a  training  evaluation  experiment  that  compared  Active  Army  and 
Reserve  Compone  personnel  using  TEC  instructional  programs  with  similar 
personnel  receiving  conventional  instruction  in  the  same  content  areas. 

The  tests  were  administered  to  personnel  in  both  Active  Army  and  Reserve 
Component  units  as  part  of  a  controlled  experiment  analyzed  in  a  separate 
report . 
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Taylor,  John  £.,  Suchman,  J.R.,  Helchlng,  W.H.,  and  Blalek,  H.M.  Development 
of  an  Individual  extension  training  system  for  managing  and  conducting  train¬ 
ing  In  the  Army  unit.  TR-77-A8.  Alexandria,  VA:  Army  Research  Institute 
for  the  Behavioral  and  Social  Science,  October  1977. 


This  report  documents  the  first  year  of  work*  to  develop  a  performance 
based  system  for  the  conduct  of  individual  training  and  evaluation  in  Army 
combat  units.  The  work  was  undertaken  in  three  phases:  (1)  Background 
information  was  gathered  and  prototype  extension  learning  packages  developed; 

(2)  a  model  of  an  Individual  Extension  Training  System  (IETS)  was  designed; 

(3)  the  prototype  IETS  was  field  tested  with  units  of  the  7th  Infantry 
Division,  Fort  Ord,  California. 

The  authors  conclude  that  it  is  feasible  to  develop  a  viable  individual 
extension  training  system  that  will  mesh  with  the  Enlisted  Personnel  Management 
System  (EPMS)  and  Skill  Qualification  Test  (SQT) . 


*See  Bialek  et  al.,  1978  for  the  second  report  of  this  series. 
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PM  TRADE,  Interim  specification  for  training  extension  course  (TEC)  materials. 
Orlando,  FL:  U.S.  Army  Project  Manager  for  Training  Devices,  20  October  1977. 


This  specification  describes  requirements  for  the  development  and  repro¬ 
duction  of  training  kits  within  the  Training  Extension  Course  (TEC) .  As  de¬ 
scribed,  the  kits  contain  training  material  in  a  synchronized  audio-visual 
format,  printed  text  (camera-ready)  format,  and  audio-only  format,  together 
with  adjunctive  training  materials  in  graphic  format  such  as  student  checklists, 
hand-outs,  and  guides  for  TEC  lessons.  Following  contractor  development,  the 
training  kits  are  reproduced  for  use  by  Army  units  In  the  field. 

Sections  include  required  and  optional  items  for  the  kits;  a  list  of 
applicable  military  and/or  federal  standards;  requirements  for  lesson  design, 
media  selection,  other  deliverables,  and  validation  of  lesson  material;  and 
quality  assurance  provisions.  Although  entitled  "interim"  and  published  in 
1977,  these  specs  were  still  in  use,  unrevlsed,  in  the  fall  of  1980. 
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Strasel,  H.C.,  Holmgren,  J.E.  Bercos,  J.,  Shafer,  J.  and  Eaklns,  R.C. 

Training  Extension  Courses  (TEC):  Cost  and  Training  Effectiveness.  Research 
Project  Draft.  ARI,  Ft.  Benning  Field  Unit.  Alexandria,  VA:  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences,  November  1977. 
Draft. 

This  report  summarizes  an  extensive  series  of  research  efforts  on  the 
TEC  program.  This  program  included  detailed  research  into  the  training 
effectiveness  of  TEC,  the  current  and  programmed  costs  and  usage  of  TEC  In 
the  Active  Army  and  Reserve  Components,  and  an  analytic  examination  of  the 
cost  effectiveness  of  TEC. 

The  principal  stated  finding  is  that  "TEC  is  both  cost  and  training 
effective  at  the  current  levels  of  training  effectiveness  and  the  current 
level  of  usage.  Analyses  further  indicate  that  if  TEC  usage  is  raised  through 
command  emphasis  and  development  of  better  methods  and  strategies  for  TEC 
implementation,  the  cost  and  training  effectiveness  of  TEC  will  be  greatly 
increased." 


Both  analytical  and  empirical  approaches  to  research  were  used.  The 
empirical  approaches  included: 

An  experimental  study  of  the  effectiveness  of  TEC  training  and 
the  retention  of  learning  with  TEC  in  comparison  to  conventional 
training  and  no  training,  for  both  Active  and  Reserve  Components. 

~  A  survey  of  TEC  usage  and  the  attitudes  and  problems  expressed 
by  TEC  users,  non-users.  Training  officers  and  NCOs,  and  Training 
Support  personnel,  in  CONUS  and  USAREUR  Active  units  and  Reserve 
Units. 


An  analysis  of  the  relationship  of  TEC  lesson  usage  and  SQT 
performance  in  six  Battalions  in  CONUS. 


-  Collection  of  all  identifiable  costs  related  to  previous  TEC 
program  and  projected  costs  on  a  uniform  annual  cost  basis  for 
the  next  ten  years. 


Collection  from  TRADOC  Schools  of  ways  TEC  has  been  u6ed  to 
supplement  or  substitute  for  conventional  training  and  the  cost 
savings  therefrom. 


Using  these  and  other  data,  a  series  of  analyses  were  performed  to  deter¬ 
mine  the  potential  and  real  cost  effectiveness  of  TEC  training  for  the  U.S, 

Army.  & 


*  / 


A  study  of  application  of  TEC  material  at  six  TRADOC  schools  showed  cost 
avings  when  TEC  was  used  in  six  cases  of  institutional  training,  tost  swoidanec 
n  25  cases  of  usage  in  self-paced  instruction  but  actual  cost  to  the  Army  in 
2  cases  of  use  in  the  remedial  or  supplemer  tel  mode.  An 
voided  in  FY  77  was  estimated  at  $3.1  million. 


Table  1 


TEC  LESSONS  SELECTED  FOE  INCLUSION  IN  THE  STUDY 


Lesson 

Identification 

Number 

Lesson  and 

Category  Title 

Applicable  U.S. 
Army  Combat  Arm 

TEC  Status  Level 
(1  January  1977) 

Tube  Artillery 

Field  Artillery 

041-061-6101-F 

Test  of  the 

Gunner’s  Quadrant, 
Part  1 

3 

041-061-6102-F 

Test  of  the 

Gunner's  Quadrant, 
Part  2 

3 

M60  Machine gun 

All  Soldiers 

94 1-07 1-00 78-F 

The  M60 

Machlnegun: 

Mechanical 

Training, 

Part  1 

1 

941-071-0079-F 

The  M60 

Machlnegun: 

Mechanical 

Training, 

Part  2 

4 

941-071-0080-F 

The  M60 

Machlnegun 

Mechanical 

Training, 

Part  3 

1 

941-071-0084-F 

The  M60 

Machlnegun: 

Firing  and 

Zeroing 

1 
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Table  2 


PERFORMANCE  TESTS  WITH  ASSOCIATED  PERFORMANCE  MEASURE  GROUPS 


Performance  Test: 

Performance  Measure  Groups: 

M60  Machine gun 

(for  use  by  all  soldiers) 

1. 

Demonstrate  changing  the  barrel 
of  the  M60  machinegun  to  prevent 
overheating. 

2. 

Demonstrate  the  3  assault  positions 
for  firing  an  M60  machinegun. 

3. 

Assume  correct  position  and  grip 
for  firing  a  bipod  emplaced  M60 
mach  inegun . 

4. 

Zero  a  bipod  mounted  M60  machinegun 
on  target. 

5. 

Clear  ammunition  from  an  M60  machine- 
gun. 

6. 

Perform  a  general  disassembly  of  the 
M60  machinegun  into  its  8  groups. 

7. 

Perform  assembly  of  an  M60  machine- 
gun  so  that  it  functions  properly. 

Target  Engagement, 

M551  Tank  (for  use  by 

1. 

Aligning  the  missile  reticle. 

Armor  branch  only) 

2. 

Aligning  the  target  and  gun/launcher 
reticle. 

3. 

Aligning  the  target  and  coax  machine- 
gun  reticle. 
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Hoyt,  W.G.,  Bennlk,  F.D.,  and  Butler,  A.K.  The  effectiveness  of  alternative 
media  In  conjunction  with  TEC  for  Improving  performance  In  MOS  related  tasks. 
TR-77-A20.  Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Science,  December  1977, 


One  of  the  series  of  related  work  conducted  for  U.S.  Army  Field  Artillery 
Schools  (USAFAS),  this  report  addresses  two  research  questions:  (1)  Does 
computer-assisted  instruction  (CAI)  provide  a  suitable  and  acceptable  media 
(sic)  for  delivering  TEC  materials  to  field  units?  and  (2)  Can  Army  lesson 
developers  feasibly  be  trained  to  convert  self-paced,  audio  visual  material 
into  CAI  format  and  easily  update  6uch  material? 

TEC  materials  were  converted  to  CAI  format  as  the  first  step.  The  job 
sequence  for  each  task  was  followed.  New  topics  were  introduced  as  needed. 

The  research  report  stated  that  this  "provided  a  logically-structured,  inte¬ 
grated,  functional,  product-oriented,  learner -centered  approach"  -  a  formal 
ISD  process. 

Four  USAFAS  course  development  personnel  were  taught  to  convert  TEC 
audio  visual  lessons-to  CAI.  Both  6ets  of  lessons  had  the  same  training 
objectives.  Four  hundred  and  ninety  hours  were  required  for  learning  the 
conversion  process.  Two  TEC  lessons  were  converted  during  the  training 
period  and  four  more  during  the  next  six  weeks.  The  six  CAI  lessons  on 
Observed  Fire  were  accompanied  by  computeruzed  course  listings,  test  scoring, 
student  records,  and  other  course  development  administration.  An  evaluation 
and  review  was  conducted  by  subject  matter  experts  (SME)  and  Army  students. 

The  costs  were  reported  as  follows: 


Course  developer  111  hours 

SMT  (2  Q  lh  hours)  3  hours 

Students  (7  @  2  hours)  14  hours 

Computer  $381 

Telephone  $300 


Therefore  they  estimate  128  man-hours  and  $681  for  each  CAI  lesson.  Working 
independently  the  contract  research  team  developed  print  media  lesson  material 
for  the  same  six  lessons.  This  provided  three  media  (and  combinations)  to  the 
Field  Artillery  School  for  future  tests  —  TEC,  CAI,  paper. 

Although  not  a  formal  COEA,  it  was  concluded  that  CAI  is  low  cost;  develop' 
men t  and  evaluation  lead  time,  short;  and  Army  developers  can  be  trained  to  con' 
vert  TEC  lessons.  It  is  suggested  expensive,  dedicated  CAI  systems  may  not  be 
necessary  but  deployed,  time-shared,  mini,  and  tactical  computers  be  used. 
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Blalek,  H.M. ,  Taylor,  J.E.,  Melching,  W.  H.,  Hiller,  J.H.,  and  Bloom,  R.  D. 
Continuation  of  development  of  an  Individual  extension  training  system  for 
managing  and  conducting  training  In  the  Army  unit.  xR-78-BlT  Alexandria, 

VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 
January  1978. 


The  purpose  of  research  reported  In  this  document  was  design  of  a  manage¬ 
ment  tool  —  an  Individual  extension  training  system  (IETS)  combined  with  the 
CATB  -  developed  battalion  management  program  (BMP) .  The  components  are 
management  procedures,  record  keeping  techniques  and  forms,  packages  (modules) 
of  task  training  materials  and  performance  checkouts  for  developing  and 
assessing  soldier  skill  proficiency,  and  guidance  for  trainers  and  training 
supervisors  in  operation  of  IETS.  Compatible  with  SQT,  EPMS,  SM,  TEC,  and 
Army  Correspondence  Program  (ACP) . 

Training  task  packages (TTP)  for  each  duty  position  refer  to  TEC,  FM, 
and  Army  Correspondence  Courses  (ACC)  as  "related  sources"  to  use.  In  this 
system  use  of  TEC  lessons  is  up  to  Individual  initiative  and  so  is  record 
keep ing .  However,  for  some  tasks,  TEC  is  the  only  training  aid  listed. 

Tills  system  would  have  provided  excellent  historical  data  for  6tudy 
of  TEC  utilization  but  it  Is  believed  not  to  have  been  implemented. 
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Smillle,  R.J.,  Keller,  J.M. ,  Setzler,  H.H.  Task  Report  —  Training  Extension 
Courses:  Development  of  a  TEC  Manager’s  Guide.  Ft.  Benning,  GA:  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences,  January  1978. 

This  Guidebook  Is  a  job  performance  aid  for  Training  Extension  Course 
(TEC)  managers  and  developers.  As  a  source  reference  it  provides  information 
on  the  organization,  management,  and  supervision  of  a  TEC  prograx;  and  the 
preparation,  development,  validation,  and  dissemination  of  TEC  lessons.  For 
other  personnel  involved  with  TEC,  the  Guidebook  serves  as  a  comprehensive 
overview  of  the  TEC  development  process. 

The  Guide,  extensively  indexed  and  illustrated  with  graphics,  is  available 
as  the  TEC  Manager's  Guidebook  from  the  U.S.  Army  Training  Support  Center,  Ft. 
Eu6tls,  VA. 
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Bennlk,  F.D.,  Hoyt,  W.G.,  and  Butler,  A.K.  Determining  TEC  Media  Alternative 
for  Field  Artillery  Individual-Collective  Training  In  the  FY78-83  Period. 
TR-78-A3.  A1exandria,  VA:  U.S.  Army  Institute  for  the  Behavioral  and  Social 
Sciences,  February  1978. 

This  report  Is  the  second  of  the  reports  dealing  with  TEC  conducted  for 
the  U.S.  Army  Field  Artillery  School  (USAFAS) .  This  particular  document 
addresses  planning  for  the  utilization  of  TEC  media  during  the  period  FY78-83. 

As  background,  the  authors  note  that  to  date  TEC  exportable  training  ha6  meant 
a  narrow  range  of  delivery  systems  —  audio  visual  filmstrips  for  the  Beseler- 
Cue-See,  printed  materials,  and  audio-directed  practice  cassettes  for  use  in 
fixed  or  portable  tape  players.  Therefore  they  have  expanded  TEC  to  include 
more  innovative  delivery  systems  such  as  computer-mediated  training  support; 
simulation  devices  for  training  and  evaluation  in  marksmanship/gunnery  and  fire 
direction;  tactical  games,  command  staff  simulation  exercises,  and  two-sided 
engagement  simulations. 

Army  training  doctrine  and  developments  seen  affecting  resource  selection 
and  utilization  in  FY  78-80  are  identified  as  are  USAFAS  training  support  situ¬ 
ations.  The  research  group  designed  a  preliminary  delivery  systems  selection 
model  and  identify  delivery  systems  actually  or  potentially  available.  Finally, 
they  provide  a  plan  for  export  and  evaluation  of  USAFAS-produced  computer- 
assisted  instruction  lessons. 

The  major  findings  of  the  research  are  as  follows: 

1.  A  need  exists  for:  (1)  closer  attention  to  soldier  characteristics; 
(2)  increased  realism  of  delivery  system  components;  (3)  selection 
of  techniques  less  demanding  of  costly  resources;  and  (4)  closer 
integration  in  the  choice  of  training  delivery  systems. 

2.  More  emphasis  on  personnel  considerations,  integrating  them  in  life 
cycle  management. 

3.  Consideration  in  FY  78-83  of  training  systems  embedded  in  fielded 
weapon  systems;  establishment  of  data  files  containing  characteristics, 
operational  status,  accessibility,  and  constraints  of  training  delivery 
systems. 


Sassone,  P.  G.  Task  Extension  Course  Research;  Literature  review,  cost  and 
training  effectiveness.  Ft.  Benning,  GA:  Litton  Mellonics  System  Development 
Division,  July  1978. 


Related  earlier  TEC  reports  in  this  series  focused  on  the  cost  and  training 
effectiveness  of  TEC,  with  effectiveness  as  function  of  individual  performance 
on  selected  hands-on  performance  tests,  and  the  relation  of  TEC  usage  to  indi- 
viduals'  performances  on  their  SQTs. 

This  report  documents  the  conduct  and  findings  of  a  training  cost  and 
effectiveness  literature  search  and  review  of  selected  literature;  it  serves 
as  the  introduction  to  the  subsequent  research  to  develop  a  TEC  Cost  and  Training 
Effectiveness  (CTEA)  methodology,  where  effectiveness  will  be  a  function  of 
units*  performance  (squads,  sections,  crews)  on  selected  missions  and  tasks 
in  their  ARTEP  evaluation  (Army  Training  and  Evaluation  Program) . 

Five  methodology  requirements  for  a  TEC  CTEA  are  specified;  and  it  is 
suggested  that  the  methods  of  microeconomics  and  welfare  economics  can  be  u6ed 
to  approach  the  methodological  requirements.  The  logic  of  a  TEC  Cost  Training 
Effectiveness  Analysis  (CTEA)  methodology  under  these  precepts  is  discussed. 
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Benesch,  M.  A.,  Bennik,  F.  D.,  Butler,  A.K.  and  Silver,  L.  A.  TEC  media 
alternatives  for  the  FY  78-83  period;  MOS  13F/FIST  Sample  Application.  P-78-5. 
Alexandria,  VA.  U.S.  Army  Institute  for  the  Behavioral  and  Social  Sciences, 
October  1978. 


This  report  Is  a  demonstration  or  application  of  the  method  for  delivery 
systems  selection  described  in  Butler  et  al.  (1978)  together  with  the  delivery 
systems  data  based  described  in  Silver  et  al.  (1978).  The  example  uses  tasks 
and  personnel  data  from  MOS  13F  FIST  (Fire  Support  Team) . 

The  "walk-through"  example  is  designed  to  be  realistic  as  possible,  A 
selected  sec  of  training  requirements  is  applied  to  the  procedure  of  determining 
a  delivery  system  mix  for  the  training  program. 
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Dept,  of  Army,  Training  Acquisition  Handbook.  Ft.  Monmouth,  N.J.:  U.S.  Army 
Communication  Research  and  Development  Command,  1  November  1978. 


The  Training  Acquisition  Handbook  was  prepared  by  CORADCOM  to  provide  a 
consolidated  set  of  information  and  guidelines  covering  training  requirements 
in  the  Army  weapon  system  acquisition  process.  It  was  prepared  under  the 
joint  auspices  of  TRADOC  and  DARCOM. 

Consideration  focus  is  placed  on  acquisition  of  extension  training  mate¬ 
rials  (ETM) ,  treated  as  a  part  of  acquisition  of  skill  performance  aids  (SPA) . 

SPA  are  defined  as  an  integrated  package  of  technical  documentation  and  training 
to  provide  the  soldier  with  information  and  skills  required  for  on-the-job 
performance. 

Components  of  ETM  are  training  managers  guides  (TMG) ,  student  guides  (SG) , 
lesson  administrative  instructions  (LAI) ,  student  lesson  sheets.  (SLS)  and 
lesson  content  materials.  The  lesson  content  materials  are  offered  in  several 
media  options  —  audiovisual,  written,  audio,  computer  mediated  (CAI,  CMI)  and 
training  devices  and  simulators.  It  is  here  that  TEC  fits  in.  The  lesson  formats 
are  generally  modeled  in  a  TEC  format  which  provides  for  three  instructional  media: 
audiovisual,  written,  and  audio.  This  guidebook  holds  that  materials  in  written 
form  are  preferred  and  that  use  of  other  formats  must  be  justified  by  a  media 
analysis.  A  depiction  of  their  process  is  shown  in  the  accompanying  figure.  A 
need  for  TEC  is  discovered  during  the  learning  strategy  determination  which 
decides  on  the  presentation  media  for  each  lesson. 

The  authors  note  that  the  ETM  process  can  also  be  applied  to  fielded  sys¬ 
tems.  Specifications  for  detailed  TEC  requirements  are  stated  to  be  contained 
in  MIL-M-63040,  Manuals,  Technical:  Extension  Training  Materials  for  Integrated 
Technical  Documentation  and  Training  (ITDT) ,  1  May  1977. 
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FiC.  1  SKILL  PERFORMANCE  AIDS  DEVELOPMENT  PROCESS 


* 


Development  of  TEC  lessons  at  this  point 


Tabic  2-2.  Representative  Instructional  Media 


Instructor  with  Standard  Aids 

1. 

Instructor /Training  Supervisor 

2. 

Charts  and  Display  Boards 

J. 

Overhead  Transparencies 

1  Printed  Materials 

i. 

Standard  Printed  Materials 

2. 

Programmed  Instruction  Texts 

3. 

Microform 

Audio  Visual 

1. 

Audio  Tapes 

2. 

Slides  and  Sound-Slides 

3. 

Filmstrips  and  Sound-Filmstrips 

A. 

Motion  Pictures  and  Sound  Motion  Pictures 

5. 

Television  and  Video  Recordings 

Training  Devices  and  Simulators 

1. 

Teaching  Machines 

2. 

Models  and  Mock-ups 

3. 

Hardware  Simulator-Trainers 

A. 

Actual  Objects 

Computer  Mediated  Training  Support  1 

i. 

Computer  Managed  Instruction  (CMI) 

2. 

Computer  Assisted  Instruction  (CAI) 

3. 

Computer-Based  Team  Training 
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Butler,  A.  K.,  Bennik,  F.  D.,  Benesch,  M.  A.  and  Sliver,  L.  A.  TEC  media  alterna¬ 
tives  for  the  FY  78-83  period:  procedure  guide  for  delivery  systems  selection. 
P-78-3.  Alexandria,  VA.  U.S.  Army  Institute  for  the  Behavioral  and  Social 
Sciences,  November,  1978. 

This  volume  (Procedure  Guide  for  Delivery  Systems  Selection)  was  produced 
as  Annex  A  of  TEC  Media  Alternatives  for  the  FY  78-83  Period:  Final  Report. 

The  Procedure  Guide  is  designed  to  aid  the  developer  In  deciding  on  selecting, 
or  developing  the  best  mix  of  Army  delivery  systems  both  for  an  overall  training 
program  and  for  each  of  its  lesson  modules.  Delivery  systems  are  defined  as 
providing  the  media  and  training  management  methods  to  conduct  training  and 
evaluation  of  soldiers  who  are  to  be  the  users  of  Army  material  systems,  or  who 
require  proficiency  in  the  performance  of  tasks  within  a  specific  MOS. 

The  developers  take  into  consideration  the  influence  and  impact  of  trainee 
requirements  and  subject  matter  characteristics  on  the  decision  making  process 
for  delivery  system  selection.  In  addition,  the  effect  of  the  settings  where 
initial  training  and  sustaining  skills  practice  occur  and  the  relation  of 
setting  exportability  of  delivery  systems  for  use  at  unit  levels  are  provided 
within  the  procedure.  Other  factors  that  the  procedure  addresses  are  the  over¬ 
all  requirements  and  constraints  such  as  type  of  program,  lead  time,  training 
method  and  funds  available. 

An  overview  of  the  procedure  is  shown  in  Figure  1,  showing  major  inputs 
and  results  for  the  two  major  procedural  blocks.  The  report  also  includes  a 
flow  chart  for  each  major  step  and  substep  in  each  of  the  procedure  blocks 
(Training  Program  Mix,  Module  Lessons  Mix) .  Various  guidelines  and  checklists 
have  been  drawn  up  to  assist  the  user  in  applying  the  procedure. 

Block  I  in  the  procedure  is  used  to  select  a  mix  of  delivery  systems  that 
meet  the  major  training  program  requirements  and  constraints.  The  procedure 
directs  the  user  to  consult  the  Delivery  Systems  Data  Base  (Silver  et  al., 

1978)  at  appropriate  points  to  determine  delivery  system  candidates.  From  this 
candidate  pool,  the  specific  delivery  systems  for  the  program  are  selected  and 
the  rationale  for  their  selection  is  prepared.  The  delivery  systems  are  currently 
available  or  will  be  available  in  the  near  future  (FY  78-83) . 

The  function  of  Block  II  is  to  assist  the  user  in  determining  the  best 
delivery  system  mix  for  individual  module/lesson  instructional  materials.  Block 
II  starts  with  the  products  of  the  design  phase:  objectives,  lesson  design 
approaches  (LDAs) ,  and  module  objective  sequence  strings.  The  user  reviews  the 
selected  delivery  systems  for  the  training  program  for  Block  I  and  from  this 
pool  assigns  the  best  delivery  system  mix  for  each  individual  model/lesson. 

Guidance  is  given  on  preparing  a  rationale  on  the  particular  selection  for  the 
lessons  in  a  module. 
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Needs/Reqts. 

Analysis 

Phase 

Decisions 


a  Job/MOS 
Structure 

•  Density 

•  Aptitudes 


Trainee 
[Population 
Data 


FEA  Job  Task 
Analysis  Phas^ 
Products 


Task  List 
Job  Task 
Data 


Delivery 
System 
Data  Base 


Block  I 


Determine 
Delivery  System 
Mix  for  Training 
Program 


Delivery 
System 
Data  Base 


Allocate  Delivery 
Systems  to  Pro¬ 
gram  Modules/ 
Lessons 


Training 
Program 
Selection 
Rational 


Design 
Phase 
Products 


Objectives 
Draft  Lesson 
Design  Approach 
Module 
Objective 
Sequence 


Module/Lesson 
Media 
Selection 
Rat  Iona  le 


Figure  1.  Delivery  System  Selection  Procedure 
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Silver,  L.  A.,  Bennlk,  F.  D. ,  Butler,  A.  K.  and  Benesch,  M.  A.  TEC  media 
alternatives  for  the  FY  78-83  period;  delivery  systems  data  base.  P-78-4. 
Alexandria,  VA.  U.S.  Army  Institute  for  the  Behavioral  and  Social  Sciences, 
December  1978. 


The  457  page  volume  describes  a  delivery  system  data  base.  It  contains 
extensive  information  on  a  wide  range  of  Army  delivery  systems  either  currently 
available  to  USAFAS  and/or  Field  Artillery  units,  or  projected  to  be  available 
from  DA  sources  in  the  FY  78-83  period.  It  was  produced  as  Annex  B  of  TEC 
Media  Alternatives  for  the  FY  78-83  Period. 


The  Delivery  Systems  Data  Base  is  designed  to  be  used  in  conjunction  with 
key  decision  points  of  the  two-Btage  Procedure  Guide  for  Delivery  System  Selec¬ 
tion  (Butler  et  al.,  1978).  The  Data  Base  may  also  be  used  independently  as  a 
source  document.  An  example  of  the  use  of  the  Procedure  Guide  together  with 
the  Data  Base  (ARI  P-78-4),  using  selected  tasks  and  personnel  data  from  MOS 
13F/FIST  (Fire  Support  Team)  is  reported  in  MOS  13F/FIST  Sample  Application 
(Benesch  et  al.,  1978). 

The  data  base  is  organized  into  "Family"  groups,  each  Family  containing 
one  or  more  "Member"  delivery  system.  A  description  of  the  Family  precedes 
its  collection  of  Member  files.  Accompanying  the  Family  description  is  a 
Member  Characteristics  table  which  summarizes  general  information  about  each 
member  that  is  contained  in  the  Family  file.  The  data  base  contains  Information 
on  42  members.  The  delivery  system  member  files  are  grouped  into  12  Families. 
The  Families  and  Members  are  as  follows: 

A.  JOB  MATERIALS 

1 .  Field  Manuals  (FMs) 

2.  Technical  Manual  (TMs) 

3.  ITDT  Technical  Manuals 

B.  PRINTED  MATERIALS 

1.  TEC  Print 

2.  Correspondence  Courses 

C.  TRANING/COMBAT  LITERATURE 

1.  Soldier' 8  Manual/Job  Book 

2.  Skill  Qualification  Tests  (SQT) 

3.  Army  Training  and  Evaluation  Program  (ARTEP) 

D.  INSTRUCTOR  WITH  STANDARD  AIDS 

1.  Class  Packets 

2.  Charts/Display  Boards 

3.  Overhead/Transparencies 

4.  Models/Mockups 


AUDIO-ONLY 


1 .  TEC  Aud io-Only 

2.  Language  Labs  (GEL) 

F.  AUDIO  VISUAL 

1.  TEC  Audio /Visual  (BESELER  CUE/SEE) 

2.  Slides/Sound  S1 ; >es 

3.  Army  Training  Films 

G.  TELEVISION/VIDEO  RECORDING 

1.  Classroom  Closed-Circuit  T^evision  (CCTV) 

2.  Television  Trainer  (TVT) 

3.  Video  Disc 

H.  COMPUTER  ASSISTED /MANAGED  INSTRUCTION  (CAI/CMI) 

1.  Remote-Access  PLANIT 

2.  PLATO  IV /TUTOR 

3.  ABACUS  Computerized  Training  System  (CTS) 

I.  EMBEDDED  TRAINING  (ET) 

1.  Operational  TACFIRE  PLANIT 

2.  TACFIRE  Training  System  (TIS) 

3.  TACFIRE  Subsystem  Team  Training  (TSTT) 

J.  TRAINING  DEVICES/SIMULATORS 

1.  Sand  Table 

2.  Fire  Control  Simulator  BT-33 

3.  Observed  Fire  Trainer  (OFT) 

4.  Artillery  Direct  Fire  Trainer  (ADFT) 

5.  M-31  Field  Artillery  Trainer 

K.  TACTICAL  ENGAGEMENT  SIMULATIONS  (TES) 

1.  SCOPES 

2.  REALTRAIN  (Exercise  for  Combined  Arms  Elements) 

3.  MILES 

L.  COMMAND/STAFF  BATTLE  SIMULATIONS 

1.  Tactical  Exercise  Without  Troops  (TEWTS) 

2.  CPX  Simulation  Facility 

3.  CAMPS  (Computer  Assisted  Map  Maneuver  System) 

4.  Combined  Arms  Tactical  Training  Simulator  (CATTS) 

5.  FIRE  FIGHT 

6 .  DUNN  KEMPF 

7.  PEGASUS 

8.  First  Battle 


A-39 


"TEC  Lessons"  in  the  general  usage  of  the  term  is  one  member  of  Family  F, 
Audio  Visual.  Called  TEC  A/V  (Beseler  Cue-See),  thi6  member  is  characterized 
as  suitable  for  individual  or  collective  use,  most  levels  of  task  training, 
institutional  and  unit  setting,  currently  available,  of  low  acquisition  costs, 
and  having  both  visual  and  auditory  presentation.  The  complete  and  detailed 
description  of  TEC  A/V  as  contained  in  this  data  base  follows  as  Figure  1. 
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FAMILY: 

MEMBER: 


F 


T.l 

T-l 


t.l, 


FAMIIY:  AUDIO  VISUAL 


MEMBER:  TEC  Audio/Visual  (BESELER  CUE/SEE) 


SOURCE/PROPONENT :  Director,  Course  Development  &  Training,  USAFAS. 

Initial  Issue:  TRADGC  Training  Material  Support  Detach.,  Tobyhanna,  ^A. 
Copies:  Chief,  Production  &  Dist.,  Tmg  Programs  Directorate,  ATSC, 
DESCRIPTION:  Ft.  Eustis,  VA. 


TEC  audlo/visual  consists  of  continuous  loop  8mm  color  film  strip 
driven  by  an  audio  cassette.  One  lesson  consists  of  one  8mm  film 
strip;  one  audio  cassette;  one  student  instruction  sheet  pasted  to 
inside  cover  of  a  gray  plastic  box  which  houses  the  lesson.  This 
type  of  lesson  is  designed  to  play/operate  with  the  Beseler  Cue/See 
viewing  machine.  Lesson  may  also  include  adjunctive  material  such 
as  TMs,  FMs ,  and  blank  forms  which  supporc  the  lesson  by  providing 
hands-on  application.  Designed  to  prepare  student  for  SQT  and/or 
ARTEP.  It  is  usable  virtually  at  any  location  with  a  power  source 
compatible  with  the  Beseler  Cue/See.  Currently  in  use  at  Active, 
Reserve,  and  National  Guard  units,  USAR  schools,  and  ROTC  schools. 
While  only  one  Instructor  is  required  to  run  the  media-paced  instruc¬ 
tion,  a  staff  must  be  maintained  to  update  programmed  training 
materials. 

Adaptation  to  an  individual  student's  needs  is  provided  by  use  of  the 
Cue/See' s  pause-proceed  system.  With  it,  the  film  and  audio  instruc¬ 
tion  are  stopped  by  a  programmed  signal  to  allow  the  student  time 
for  "hands-on"  skill  practice  exerclsrs  or  written  responses  to 
questions.  The  student  can  reactivate  the  program  at  will  by  merely 
(pressing  the  Cue/See  proceed  button.  By  using  a  "Responder" 
accessory,  multiple-choice  questions  can  be  presented  during  the  pro¬ 
grammed  pause.  The  program  will  then  resume  only  after  the  correct 
(answer  is  given. 

TRAINING  AND  EVALUATION  APPLICATIONS: 


X  Individual 


COMMENTS 

Each  TEC  lesson  is  keyed  to  a  task  6r  teak  grour  in 
Soldier's  Manuals.  Group  projection  &  audio,  or 
headphones  for  private  self-studv. 

X  Collective  Small-group  projection  &  audio  permits  teamwork  in 
solving  problems,  e.e.,  leadership  workshop . _ 


Fijnre  1. 


TF.C  A/V  (Beseler  Cue- See)  as  described  in  the  Delivery 
Svst  er>  Data  Base 
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FAMILY: 

MEMBER: 


F 


F.l.4.2 


r .1 .4.3 


RESPONSE: 

a.  Verbal/ Symbolic 

X  Choice  Selection 
X  Specific  Recall 
X  Composed /Created 

b.  Performance 

_  Indication 

_  Manipulation 

X  Read/ Interpret 
X  Lis  ten/ Interpret 

_  Voice  Composition 

X  Situation  Evaluation 
X  Decide  Action 

TRAINING  MANAGEMENT: 

a.  Feedback 

X  Informative 
X  Evaluative 
X  Corrective 
Summary 
X  Immediate 
X  Delayed 

b.  Presentation  Control 

X  Learner 
X  Media 

Instructor 


COMMENTS 

Verbal,  written,  and/or  performance, 
as  programmed  in  the  lesson  -  lends 
Itself  to  multiple-choice. 


Depends  on  performance  task. 
Performance  must  be  compared  with 
lesson  or  checked  by  observer. 


Student  can  press  "proceed"  button  cn^ 
Only  if  student  attempts  response.  j 


Push  proceed  button  to  advance. 
Built-in,  pace  designed  for  self- 
taught  lesson. 
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FAMILY: 

MEMBER: 


5 

5.1 


TRAINING  MANAGEMENT  (CONI' D) 

COMMENTS 

c.  Sequencing 

X  Linear  _ 

_  Response  Branching  _____________________________ 

_  History  Branching  • 

X  Repetitive  Practice  May  be  given  a  series  of  problems. 

d.  Recordkeeping 


_ _ Automatic  (Temporary  Storage) 

_  Automatic  (Permanent  Storage) 

X  Manual  Requlrcd/optional  per  Trn; 

JL_  None  _ 1 _ 


Mgmt.  Plan. 


SETTING: 


X  INSTITUTIONAL 


x  Service  School  See  P.1.5.3. _ 

_ Army  Training  Ctr  (ATC)  __________________ 

_  Div/Post  Installation 

Schools  _ 

X _ UNIT  Active,  Reserve,  and  National  Guard  units. 


_  In-System  (Embedded) 

X  Job-Station  ^cc  F«l»5.3. 

X  Individual  Learning  Ctr  (HC) 

X  Barracks /Home  ___________ 

X  Garrison  Training  Area  ____________ 

X _ Local  Training  Area  ____________ 

_ Major  Training  ____________ 

_ _ National  Range  _ _ _ 


FAMILY: 

MEMBER: 


F.1.5. 


REQUIREMENTS :  COMMENTS 

Power  source  compatible  with  the  Beseler  Cue/See:  105-130  voit« 
AC.  60Hz,  30-100  watts.  Use  In  field  with  portable  electric 


generator. 


F.1.6  EXPORTABILITY: 


X  In  Units  _ Planning  _ No 

F. 1.6.1  INITIAL  OPERATIONAL 
CAPABILITY  DATE: 


F.1.6. 2 


X  Currently  Available 

_  Future  Date 

_  Unknown 

METHOD: 


F.1.6. 3 


X  Mail/Ship 

_  Install 

_  Remote  Access 

_  In  System 

_  Job  Materials 

BASIS  OF  ISSUE: 


TRADOC,  Trng.  Material  Support  Detach.,  Tobyhanna, 


(Telecommunications) 


•  1.6.4 


_  On  Demand 

X  Unit  Issue 

SOURCE: 

X  Known 

_  Prjoable 

Unknown 


Per  distribution  plan.  8  Cue/See  devices  and  4 
viewing  screens  per  battalion. _ 

Issue:  TRADOC  Trng.  Materials  Detach.,  Tobyhanna,  f 
Lessons:  Chief,  Prod.  6  Dist.,  Trng.  Programs 

Directorate,  ATSC,  Ft.  Eustls,  VA _ 


l 


i 


FAMILY: 

MEMBER: 


.1.7  SUPPORT  REQUIREMENTS: 

.1.7.1  LOGISTIC/!  SUPPORT: 

X  Storage  Areas 
X  Spare  Parts 

_  Additional  Copies  On  Hand 

_  Expendable  Material 

.1.7.2  MAINTENANCE  REQUIREMENTS: 


COMMENTS 

Beseler  Cue/See,  A/V  kits,  screens. 
Lamps  for  Beseler  Cue/See. 


X  Yes 


Provided  by  the  Trng.  Aids  Office 


_  Operator  8uj 

_  Organizational  Maintenance  __ 

_  Direct  Support  Maintenance  _ 

_  General  Support  Maintenance 

_  Depot  Maintenance 

M.7.3  MANNING  REQUIREMENTS:  Total  -  _1_ 

_  System  Operators  _ 

X  Instructors/Facilitators  1 


supporting  each  unit. 


Librarian  for  materials. 


_  Simulation/Mbnitor  Positions 

_  Support  Personnel  _ 

'.1.7.4  SCHOOL  OPERATIONAL  SUPPORT:  , 

* 

X  Special  Personnel  Skills 

_ X_  Special  Equipment 

_  Facility  J 

_  Reproduction  _j 

_ Update  _J 


Staff  of  officers/civilians  with  de¬ 
velopmental  skills  in  programed  trng. 

materials.  Standard  photographic  gear. 

Access  to  audio  cassette  recording, 

pulsing,  L  reproduction  gear  unless 

contracted  out. 
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May 6,  P.  V.,  Holmgren,  J.E.,  and  Shelnutt,  J.  B.  Current  Use,  Pattern8  of  Use 
and  Factors  Affecting  Use  of  the  Army  Training  Extension  Course  (TEC)  Program. 
TR-79-A3.  Alexandria,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences,  April  1979. 


This  report  describes  research  designed  to  determine  the  extent  of  use, 
patterns  of  use  and  factors  affecting  use  of  TEC  lessons  based  on  a  survey. 

The  survey  was  conducted  in  two  phases.  In  Phase  1,  134  selected  Active 
and  Reserve  Component  batallions  and  37  TRADOC  activities  within  CONUS  moni¬ 
tored  their  TEC  use  during  a  two-month  period.  In  these  units  a  form  provided 
by  ARI  was  completed  each  time  a  TEC  lesson  was  used. 

During  Phase  2,  3404  soldiers  and  608  unit  trainers  in  85  CONUS  and  USAREUR 
battalions  completed  questionnaires  regarding  their  TEC  use.  Battalion  level 
training  personnel  were  interviewed  in  42  of  these  battalions,  16  associated 
bridgade/division  level  training  officers  and  seven  associated  Training  Aids 
Support  Centers  (TASCs) . 

The  major  findings  were  as  follows: 

(1)  A  total  of  78,742  uses  of  TEC  were  recorded  during  Phase  1.  This 
yielded,  for  F0RSC0M  battalions,  an  average  TEC  u6e  per  man  per  month  of  .353 
lessons  in  the  Active  Component  and  .802  lessons  in  the  Reserve  Component. 

(2)  Over  90%  of  all  recorded  use  occurred  in  groups.  Most  of  this  u6e 
was  mandatory  (81.8%)  and  occurred  during  duty  hours  (96.6%). 

(3)  Over  50.2%  of  soldiers  sampled  in  Phase  2  had  used  TEC;  35.3%  had  never 
heard  of  TEC. 

(4)  Most  soldiers  (TEC  users)  ,  unit  trainers,  and  all  interviewees  felt 
that  TEC  should  be  continued.  Most  users,  trainers,  and  interviewees  also  pre¬ 
ferred  TEC  to  4  of  7  other  training  methods  (lectures,  small  group  instruction, 
training  films,  Soldier's  Manual). 

(5)  Reasons  most  often  cited  by  soldiers  for  lack  of  use  pertaineu  to 
ignorance  of  TEC,  unavailability  of  equipment  and  lack  of  encouragement  to  use. 
Least  often  cited  were  reasons  of  lack  of  need  for  or  perceived  benefit  from 
TEC  training. 

(6)  Command  emphasis  on  TEC  was  judged  to  be  small  to  moderate  at  all 
levels  and  most  often  took  the  form  of  announcements,  briefings  or  specific  orders/ 
requests. 

(7)  Most  soldiers  (74%)  learned  about  TEC  from  their  unit  trainers. 

(8)  Basis  of  issue  for  CUE-SEES  and  TEC  lessons  was  considered  adequate 
by  over  half  of  battalion  interviewees. 

(9)  Equipment  failure  did  net  ippear  to  be  a  significant  problem. 
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The  authors  of  the  report  state  that,  "Data  on  TEC  usage,  when  combined 
with  cost  and  effectiveness  data,  will  allow  determination  of  the  cost  effec¬ 
tiveness  of  the  TEC  program.  It  will  also  be  used  as  Input  to  the  development 
of  an  Implementation  plen  designed  to  Increase  cost  effectiveness  and  promote 
optimal  utilization  of  the  program." 
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Bercos,  J.  and  Eakins,  R.C.  Task  Report:  Training  Extension  Course  Research: 
Extension  of  Training  Extension  Course  Cost  and  Training  Effectiveness  Analysis 
Data  Collection.  Working  Paper.  Ft.  Benning,  GA:  Litton  Mellonlcs  System 
Development  Division.  June  1979.  Unpublished. 


This  document  contains  forms  and  implementation  instructions  developed  for 
the  collection  of  unit  training  data  required  as  input  to  the  Training  Extension 
Course  (TEC)  Cost  and  Training  Effective  Analysis  (CTEA)  Methodology. 

The  completed  data  collection  forms  were  validated  in  a  review  by  four 
active  Army  battalions  and  were  used  for  the  development  of  hypothetical  unit 
training  data. 

The  collected  data  constitutes  the  xnput  requirements  to  the  TEC  CTEA 
Methodology. 
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Hawkins,  W.  W.  and  Kribs,  H.  D.  Technology  for  an  efficient  delivery  system. 
NAVTRAEQUIPCEN  78-00129-1.  Orlando,  FL:  Naval  Training  Equipment  Center, 
June  1979. 


The  purpose  of  this  study  was  to  determine  the  feasibility,  specifications, 
and  costs  of  using  the  emerging  video  and  computer  technologies  for  advanced 
instructional  delivery  systems  (AIDS)  in  lieu  of  traditional  audiovisual  media. 

To  accomplish  these  objectives  the  research  team  surveyed  the  current  6tate-of- 
the  art  media  used  in  ISD-based  training  for  Navy  Aviation  Weapon  Systems. 

These  data  served  as  a  baseline.  A  concurrent  survey  of  technology  and  costs 
of  potential  AIDS  was  also  performed.  Third,  the  cost  of  AIDS  was  compared  to 
the  current  media. 

The  authors  found  that  if  AIDS  were  part  of  the  media  pool  available  for 
ISD  selection,  they  would  be  a  feasible,  effective  choice,  delivered  at  projected 
25-75%  developmental  cost  savings. 


Holmgren,  J.E.,  Sassone,  P.G.,  and  Bercos,  J.  Task  Report  Training  Extension 
Course  Research:  training  extension  course  cost  and  training  effectiveness 
analysis  methodology.  Ft.  Bennlng,  GA:  Litton  Mellonics  System  Development 
Division,  July  1979. 


The  research  covered  in  this  report  is  the  development  of  a  cost  and  train¬ 
ing  effectiveness  (CTEA)  methodology  to  identify  the  contributions  of  Training 
Extension  Course  (TEC)  training  to  a  unit  performance  as  measured  by  the  Army 
Training  and  Evaluation  Program. 

The  development  of  the  Methodology  required  four  areas  of  specification. 
First,  a  measure  of  unit  effectiveness  is  assumed  to  be  derivable  from  the  pass/ 
fail  (satisfactory /unsatisfactory)  scoring  criteria  currently  in  use.  Second, 
the  assumption  is  made  that  costs  of  field,  garrison,  and  TEC  training  can  be 
determined.  Third,  that  a  resource  allocation  model  can  be  developed  from  which 
can  be  inferred  the  economic  value  of  TEC.  The  effectiveness  production  func¬ 
tion  assumes  some  specifiable  relation  between  inputs  and  outputs  of  the  train¬ 
ing  process  which  operates  under  specific  budget  constraints.  Lastly,  the  use 
of  the  valuation  methodology  in  a  present  value  analysis  to  derive  an  ultimate 
dollar  value  of  TEC  is  outlined. 

The  TEC  CTEA  methodology  can  be  used  to  provide  an  objective  economic  eval¬ 
uation  of  the  contribution  TEC  training  makes  to  unit  performance.  This  valua¬ 
tion  methodology  can  also  be  applied  to  any  training  program  which  serves  a 
complementary,  rather  than  as  a  substitute,  training  program. 
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Osborne,  A.D.  Human  Factors  Research;  Training  Effectiveness  and  Training 
Extension  Courses.  Final  Report.  Ft.  Bennlng,  GA:  Litton  Mellonlcs,  July  1979. 
Unpublished. 


This  report  summarizes  research  undertaken  to  support  ongoing  ARI  programs — 
four  concerning  training  effectiveness  analysis  (TEA)  and  five  related  to  Training 
Extension  Courses  (TEC).  The  five  TEC  research  areas  covered  were  TEC  evaluation, 
TEC  validation,  a  CTEA  literature  review,  proposed  TEC  CTEA  data  collection,  and 
TEC  CTEA  methodology. 

Separate  research  reports  were  written  fully  documenting  each  subject  area. 
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Osborne*  A.  D.  Task  Report:  Training  Extension  Course  Research:  Training 
Extension  Course  Validation.  Ft.  Benning,  GA:  Litton  Mellonics,  July  1979. 
Unpublished. 


This  task  report  documents  an  Investigation  and  determination  of  the 
effectiveness  of  current  practices  and  procedures  used  by  service  schools 
to  validate  TEC  lessons.  The  research  sponsor  was  the  ARI  Field  Unit*  Ft. 

Benning,  GA. 

During  the  period  August  to  November  1978,  TEC  validations  being  conducted 
by  five  different  service  schools  were  observed. 

The  validation  guidance  contained  in  the  TEC  Manager's  Guidebook  is  used 
as  the  basis  for  planning  TEC  validations.  Several  deviations  from  the  pre¬ 
scribed  procedures  were  observed;  however,  these  adjustments  were  the  results 
of  conscious  decisions  made  with  the  intent  of  facilitating  mission  accomplish¬ 
ment  with  available  resources.  The  lack  of  subject  matter  expert  (SME)  influ¬ 
ence  during  lesson  development  and  the  failure  to  use  hands-on  performance  tests 
in  validations  have  an  adverse  impact  on  the  quality  of  completed  TEC  lessons. 

This  report  describes  shortcomings  associated  with  the  current  practices 
and  procedures  used  in  TEC  lesson  validations  and  presents  suggested  modifications. 
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Sassone,  P.  G.  and  Bercos,  J.  Task  Report  Training  Extension  Course  Research: 
Training  Extension  Course  Cost  and  Training  Effectiveness  Analysis  Methodology. 
Ft.  Benning,  GA:  Litton  Mellonics  System  Development  Division,  July  1979, 
Unpublished  Paper. 


This  report  concerns  the  development  of  a  TEC  Cost  and  Training  Effective¬ 
ness  (CTEA)  methodology.  It  covers  the  second  phase  of  research  reported  in 
Sassone  (1978)  —  A  Review  of  Selected  Training,  Cost,  and  Effectiveness 
Literature.  The  authors  outline  the  development  of  a  TEC  CTEA  methodology.  It 
is  unique  in  that  it  measures  the  cost  effectiveness  of  a  single  part  of  a  total 
unit  training  program  as  measured  by  the  AKTEP.  The  report  discusses  the  basis 
of  the  methodology  ii  economic  theory,  describes  its  adaptation  to  Army  training, 
explains  the  development  of  the  valuation  part  of  the  methodology,  presents  an 
algebraic  example,  and  shows  the  use  of  the  valuation  methodology  in  a  present 
value  analysis  to  derive  an  ultimate  dollar  value  of  TEC.  Also  discussed  is 
the  implementation  of  the  TEC  CTEA  methodology  in  terms  of  data  requirements, 
candidate  effectiveness  production  function,  and  determination  of  the  value  of 
TEC  training. 


Markham,  Roger  D.  "Immediate  learner  achievement  as  an  effect  of  aesthetic 
.'♦mbelllshment  in  education  art."  A  paper  published  In  Proceedings  of  the  First 
Interservice/Industry  Training  Equipment  Conference.  Orlando,  FL:  Naval 
Training  Equipment  Center,  November  1979. 


This  paper  reported  research  undertaken  to  determine  the  effects  of  levels 
of  artwork  in  audlo-vi6ual  sound  slide  teaching  devices  on  message  comprehension. 
An  attempt  was  made  to  find  out  whether  students  can  obtain  as  much  "information 
from  simple  art  (line  drawings,  stick  figures,  geometric  patterns,  etc.)  as  they 
can  from  a  more  complex  rendition  of  the  6ame  subject,  including  full  human 
figures,  extremely  detailed  subject  matter,  use  of  color  and  more  embellishment 
for  the  purpose  of  intensifying  the  aesthetic  quality  of  the  visual." 

This  has  economic  implications  for  the  production  of  TEC  lessons.  Markham 
notes  that  the  filmstrip  used  in  the  Besler  Cue-See  is  produced  by  photographing 
coramei  .ially  developed  art  work  and  each  lesson  on  the  average  contains  150 
pieces  of  art.  The  current  cost  of  a  simple  frame  is  $15;  a  complex  one,  $75. 

He  developed  a  simple  set  for  $10.  The  author's  research  premise  is  that  if 
a  simple  drawing  could  achieve  the  same  results  as  a  complicated  embellished 
art  frame,  the  Government  would  realize  substantial  savings  in  money  as  well 
as  in  time  invested  in  developing  and  producing  TEC  lessons. 

Using  a  simple  set  of  art  frames,  the  author  test6  the  null  hypothesis 
that  there  would  be  no  difference  in  achievement  of  subjects  who  are  taught  the 
same  concepts  using  two  sets  of  visuals  —  simple  and  complex  (see  attached 
figure).  Using  two  groups.  Army  Reservists  and  college  students,  he  found  non¬ 
significant  differences  on  a  20-item  comprehension  test.  This  work  implies 
that  dramatic  saving  in  cost  16  possible  using  simple  line  graphics  rather  than 
finished  art  for  TEC  lessons. 
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Example  of  Complex  Art  as  defined  by  ihc  l ».S.  Army  TEC  Program 


Eumplj  Pt  Sirie1^  An  uied  \,n  iHit  itudy 
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Holmgren,  J.  E. ,  ?;ye r,  F.  N.f  HllligosB,  R.  E.,  and  Heller,  F.  H.  "The 
effectiveness  of  Army  training  extension  course  lessons  on  videodisc". 
Journal  of  Educational  Technology  Systems,  Vol.  8(3),  1979-80,  pp.  263-274. 


Three  sets  of  audiovisual  lessons  from  the  Army  Training  Extension  Course 
(TEC)  Program  were  presented  to  samples  of  enlisted  bduiers  on  either  a  proto¬ 
type  videodisc  player  or  the  film  and  audio  cassette  player  currently  used  for 
TEC  lessons.  The  training  effectiveness  of  the  lessons  was  determined  by 
administering  validated  hands-on  performance  tests  to  the  trained  soldiers  and 
also  to  a  baseline  group  that  had  not  viewed  the  TEC  lessons.  Initial  results 
6how  that,  for  all  three  sets  of  lessons,  both  groups  of  trained  soldiers  per¬ 
formed  significantly  better  than  those  in  the  baseline  group.  The  difference 
between  the  two  trained  groups  is  not  statistically  significant  for  two  of  the 
three  lesson  sets;  for  the  third  set,  there  is  a  significant  difference  of  ten 
percent  in  mean  performance  test  score  in  favor  of  the  film  and  audio  cassette 
player. 

Three  sets: 

(1)  Two-lesson  set  for  Armor  unit  personnel 

(2)  Two-lesson  set  for  FA  unit  personnel 

(3)  Three-lesson  set  for  IN  unit  personnel 

Demographic  statistics;  performance  test  results  and  number  in  each 
experimental  group  are  shown: 
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Table  1.  Background  Information  on  the  Three  Samples 


Background  Statistic 

Armor 

Sample 

Artillery 

Infantry 

Mean  GT  score 

98  4 

99.8 

100.0 

Mean  years  of  education 

11.7 

11.8 

11.5 

Mean  pay  grade* 

3.9 

3.7 

3.0 

Mean  months  in  service 

44.9 

36.3 

29.1 

%  with  previous  TEC  training  in  subject  area 

18 

28 

18 

%  with  BCT  training  in  subject  area 

7 

16 

70 

%  with  AIT  training  in  subject  area 

14 

49 

74 

%  with  training  since  AIT  in  subject  area 

24 

56 

49 

*  El  through  E6  were  coded  1  through  6  and  than  averaged. 


Table  2.  Performance  Test  Results 


£  xperimen  tal  Condi  tbns 


Sample 

Standard 

Beseler 

Combined 

Videodisc 

Standard 

Videodisc 

Modified 

Videodisc 

Baseline 

X 

56.6 

46.2 

46.9 

45.5 

29.4 

Armor 

sx 

3.6 

3.3 

4.7 

4.5 

2.3 

N 

31 

32 

16 

16 

38 

Artillery 

X 

54.1 

526 

58.5 

50.5 

23.8 

5.2 

5.2 

10.2 

6.0 

4.3 

N 

33 

31 

8 

23 

37 

X 

60.6 

50.7 

49.8 

51.8 

26.8 

Infantry 

4.0 

4.1 

6.0 

5.6 

3.4 

N 

31 

32 

17 

15 

33 
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Vineberg,  R.  and  Joyner,  J.  N.,  Instructional  system  development  (ISP)  in  the 
Armed  Services:  methodology  and  application.  HumRRO-TR-80-1 .  Alexandria,  VA: 
Human  Resources  Research  Organization,  January  1980. 


This  study  examined  Instructional  System  Development  (ISD)  methodologies 
and  practices  in  the  Armed  Services  during  the  period  August  1977  -  Hay  1979. 
The  major  finding  is  that  ISD  is  not  being  used  either  to  optimize  total  system 
effectiveness  or  to  maximize  training  efficiency.  Some  findings  are  pertinent 
to  an  evaluation  of  TEC. 

The  first  is  the  area  of  media  selection. 

Findings: 

1.  Methodology  -  Procedures  for  selecting  training  media  appear 
adequate  to  match  the  presentation  (stimulus  and  response) 
requirements  of  instructional  activities  to  appropriate  media, 
if  instructional  activities  have  been  specified  in  sufficient 
detail . 

2.  Application  -  Training  media  are  not  systematically  selected  on 
the  basis  of  requirements  of  instructional  activities  (which  are 
also  not  specified). 

3.  Application  -  Developers  generally  do  not  have  freedom  to 
select  among  alternative  media.  Choices  and  changes  in  media 
are  usually  directed  by  command  policy. 


Recommenda  t ion : 


1.  Selection  of  appropriate  media  is  contingent  on  how  well  in¬ 
structional  activities  have  been  specified.  No  change  in  the 
current  models  for  matching  media  to  activities  is  appropriate 
until  activities  are  more  widely  specified,  and  these  models 
can  be  tested. 

The  second  is  in  the  area  of  review  end  selection  of  existing  material. 

Findings: 

1.  Methodology  -  The  currently  used  Army  model  states  that  decisions 
to  use  existing  materials  (rather  than  author  new  instruction) 
are  to  be  based  on  the  appropriateness  of  these  materials  to  the 
previously  opecified  characteristics  of  objectives,  methods,  and 
media. 

2.  Application  -  Characteristics  to  be  identified  in  judging  the 
appropriateness  of  existing  training  materials  are  not  specified. 

3.  Application  -  Review  and  use  of  existing  training  materials  are 
minimal ,  except  for  chose  in  a  course  that  is  being  revised. 
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Recommendation 


1.  The  specification  of  necessary  properties  of  materials  for  particular 
training  situations*  and  the  description  and  cataloging  of  existing 
materials  to  permit  the  interchange  of  matching  components  across 
courses*  represent  a  degree  of  perfection  that  is  not  presently 
attainable.  An  attempt  to  reduce  the  review  and  selection  of  existing 
materials  to  a  systematic  procedure  is  to  act  as  if  the  methods  of  a 
well-developed  technology  were  available  in  an  area  in  which  judgment 
is  in  fact  the  dominant  factor.  No  change  in  present  practice  is 
recommended. 
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APPENDIX  C 


ABBREVIATIONS  AND  ACRONYMS 


ABBREVIATIONS-  AND  ACRONYMS 


AD 

AMSAA 

AP 

AR 

ARI 

ARTEP 
AS  ARC 
ATM 
ATSC 

BDM/CARAF 

CAFTDR 

CAI 

CD 

C/E 

CHRT 

CMI 

COEA 

CONUS 

CORADCOM 

CTEA 

DA 

DARCOM 

DAS 

DCP 

DEVA  IPR 
DRIMS 
DSARC 
DT 

DT/OT 


Advanced  Development 

Army  Materiel  Systems  Analysis  Agency 

Acquisition  Plan 

Amy  Regulation 

U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences 
Army  Training  and  Evaluation  Program 
Army  Systems  Acquisition  Review  Council 
Analogous  Task  Method 
Army  Training  Support  Center 

The  BDM  Service  Company/ Combined  Anns  Research  and 
Analysis  Facility 

Commercially  Available/Fabricated  Training  Device 
Requirement 

Computer  Assisted  Instruction 
Combat  Development 
Cost  Effectiveness 

Coordinated  Hunan  Resources  Technology 
Computer  Mediated  Instruction 
Cost  and  Operational  Effectiveness  Analysis 
Continental  United  States 

Communications  Research  and  Development  Cocmand 
Cost  and  Training  Effectiveness  Analysis 
Department  of  the  Army 

U.S.  Army  Materiel  Development  and  Readiness  Command 

Defense  Audit  Service 

Decision  Coordinating  Paper 

Development  Acceptance  In-Process  Review 

Diagnostic  Rifle  Marksmanship  Simulators 

Defense  System  Acquisition  Review  Council 

Development  Test 

Development  Test/Operational  Test 
Actual  Effectiveness 


% 


Design  Effectiveness 


ETM 

FDTE 

FSED 

HRDT 

IPISD 

>  IPR 
IPS 
ISD 
ISTEA 
ITDT 
ITV 
JGD 
LAI 
LCC 
LCCE 


Extension  Training  Materials 
Force  Development  Testing  and  Experimentation 
Full-Scale  Engineering  Development 
Hunan  Resources  in  Design  Trade-Offs 
Interservice  Procedures  for  Instructional  Systems 
Development 
In-Process  Review 
Integrated  Personnel  Support 
Instructional  Systems  Development 
Initial  Screening  Training  Effectiveness  Analysis 
Integrated  Technical  Documentation  and  Training 
Improved  TOW  Vehicle 
Job  Guide  Development 
Lesson  Administrative  Instruction 
Life  Cyole  Costs 
Life  Cycle  Cost  Elements 


LCSM 

LOA 

LR 

MENS 

MHPERCEN 

MM 

MOS 

HODIA 

NET 

OAP 

OOP 

OJT 

OT 

OTEA 

PH  TRADE 

POI 

pa 

PQQPRI 


RDTE 

SAAC 

SAT 

SG 

SLS 

TABG 

TCA 

TDDA 

TDLOA 

TDLR 

TDR 

TDS 

TEA 

TEC 

TECEP 

TEEM 

TER 

TM 

TMG 

TOT 

TRA 

THDOC 

TRAINVICE 

TRAM 

TRAMOD 

TRASANA 

TSE 

TSEA 

TTS 

USAREUR 

WSTEA 


Life  Cycle  System  Management  Model 

Letter  of  Agreement 

Letter  Requirement 

Mission  Element  Needs  Statement 

Military  Personnel  Center 

Maintenance  Manpower  Modeling 

Military  Occupational  Specialty 

Method  of  Designing  Instructional  Alternatives 

New  Equipment  Training 

Outline  Acquisition  Plan 

Outline  Development  Plan 

On-the-job  Training 

Operational  Test 

Operational  Test  and  Evaluation  Agency 
Project  Manager,  Training  Devices 
Program  of  Instruction 
Petroleum,  Oils,  and  Lubricants 
Provisional  Qualitative  and  Quantitative  Personnel 
Requirements  Information 
Research,  Development,  Test  and  Evaluation 
Systems  for  Air-to-Air  Simulation 
Systems  Approach  to  Training 
Student  Guide 
Student  Lesson  Sheets 
Training  Analysis  and  Evaluation  Group 
Training  Consonance  Analysis 
Training  Developers  Decision  Aid 
Training  Device  Letter  of  Agreement 
Training  Device  Letter  Requirement 
Training  Device  Requirement 
Training  Developments  Study 
Training  Effectiveness  Analysis 
Training  Extension  Course 

Training  Effectiveness,  Cost  Effectiveness  Prediction 

Training  Efficiency  Estimation  Model 

Transfer  Effectiveness  Ratio 

Technical  Manual 

Training  Manager's  Guide 

Transfer  of  Training 

Training  Requirements  Analysis 

Training  and  Doctrine  Cocmand 

Training  Device  Effectiveness  Model 

Training  Analysis  Model 

Training  Requirements  Analysis  Model 

TRADOC  Systems  Analysis  Agency 

Total  Systmn  Evaluation 

Training  Subsystem  Effectiveness  Analysis 

Technical  Training  School 

U.S.  Amy,  Europe 

Weapon  System  Training  Effectiveness  Analysis 


